








STUDIES IN EXTRA-SENSORY PERCEPTION: 
111. A REVIEW OF ALL UNIVERSITY OF COLORADO EXPERIMENTS 


By 
Dorothy R. Martin and Frances P. Stribic 


ABSTRACT 


All data, with the exception of a short series 
designed to test for telepathy, obtained by the experi- 
menters over a three year period in the University of 
Colorado laboratory are here reported. The data embrace 
a total of 12470 runs (311,750 trials) performed by 322 
subjects. Four primary experimental series are included: 
1. a preliminary series of 3171 runs performed by 39 
subjects in 1937; 2. a group series of 2840 runs per- 
formed by 284 subjects in 1938-1939; 3. a main experi- 
mental series involving 4180 runs performed by 13 sub- 
‘jects in 1938-1939; 4. a series of 2279 runs involving 
several variations in experimental technique and per- 
formed by a single subject in 1938-1939. In all except 
the preliminary series, tests were screened. Unless 
otherwise indicated the DT or ‘down through' technique 
was used exclusively. 


A control series obtained by matching the sub- 
ject's guesses against the reverse order of the pack was 
employed throughout. This series is termed the reverse 
series. An additional control was supplied by chance ex- 
pectancies as computed upon the binomial hypothesis. 


Results for the four series yield direct scoring 
averages of 5.80, 5.00, 5.79, 6.90, with critical ratios 
of 24.09, -.12, 25.19, 43.35, respectively, as against 
reverse scoring averages of 4.98, 4.95, 4.97, 5.04, with 
critical ratios of -.40, -1.39, -.90, 1.06, respectively. 
The aggregate data yield a direct scoring average of 
5.83 with C.R. 45.70 as compared with a reverse average 
of 4.98 with C.R. 1.00. 


Detailed analysis of the data reveals: 1. pre- 
hounced variation in the scoring success achieved by 
different subjects, 16 subjects attaining averages which 
yield eritical ratios above 2.5, 315 subjects attaining 
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averages within the -2.5 to +2.5 C.R. range. A single 
subject (who performed only 30 runs) achieved a score 
with a critical ratio below -2.5. 2. marked daily fluc- 
tuation in scoring for both high and low-scoring groups, 
with notably greater consistency among the former. Cer- 
tain unusual scores suggest possible correlation between 
scoring success and physical fitness. 3. a significant 
excess for consecutive hits in the direct scoring of the 
high-scoring group. Scoring advantage would appear to 
reside primarily in this excess. 4. inconclusive results 
with respect to the relation between scoring success and 
card position. For a single subject, however, there is 
maintained a sufficiently direct relationship between 
these factors to suggest the possibility that there may 
be a general functional relationship. 5. no evidence in 
support of the implication that records of high-scoring 
'subjects' may have arisen from fortuitous confluence, 


INTRODUCTION 


In December, 1936, experimental investigations 
were undertaken at the University of Colorado with a view 
to examining the hypothesis of Extra-Sensory Perception, 
It is the purpose of this paper to present in rather 
great detail a complete resumé of all work relating to 
the clairvoyance aspect of the hypothesis. Much of this 
work has been exploratory in nature and has little of in- 
terest to offer. Certain of the more significant results 
have already been reported in this journal (7, 8, 9). It 
seems, nevertheless, desirable that the entire collection 
of data be presented at this time in order that a back- 
ground may be offered against which the more significant 
portions of the work may be evaluated. 


EXPERIMENTAL MATERIALS 


The materials used throughout all tests consisted 
of so-called standard ESP cards. These cards were ar- 
ranged in packs of twenty-five, there being in each pack 
five cards of each of five designs (star, cross, waves, 
circle, rectangle). Prior to the fall of 1938, and 
throughout all of the experimental work with our special 
subject Mr. C. J., heavy hand-stamped cards supplied 
through the courtesy of Dr. J. B. Rhine were used exclu- 
sively. These cards present few if any visual or tactual 
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cues. Subsequent to September, 1938, when an adequate 
supply of these cards was no longer available, the regu- 
lar commercial cards were substituted for all subjects 
except, as previously noted, for Mr. C.J. These cards, 
which are by no means free from visual and tactual cues, 
were used only under screened conditions. 


GENERAL PROCEDURES 


Orientation i," ip 
rior to the undertaking of the initial test, 


all subjects were informed as to the purpose of the 
tests, were given a description of test procedure, and 
were invited to examine a sample pack of ESP cards. 
Brief comment was made on the results to be expected on 
the basis of a chance hypothesis and on reported devia- 
tions from this hypothesis. Skepticism toward the ESP 
postulate was given generous expression. The need for 
extensive and rigorously controlled experimental inves- 
tigation was emphasized. 


Procedure for individual tests. 

The general procedure for individual tests is 
recorded below. Any exceptions to this procedure will 
be treated separately in subsequent sections of this re- 
port. 





The subject was seated at a table and was separ- 
ated from the experimenter by a 15"x18" opaque wooden 
Screen without apertures. The DT or down through method 
was employed throughout. The experimenter shuffled and 
cut the pack, and placed it face down before her, taking 
care to avoid seeing the bottom card. The subject then 
recorded twenty-five guesses for the down through order 
of the pack. After the completion of each run, and in 
the presence of both subject and experimenter, the pack 
Sequence was recorded and checked against the guessed 
order and the coincidences noted. The data obtained by 
the matching of the guessed order against the card order 
for which it was intended will be termed the dtrect ser- 
tes (in contrast to the reverse series---a control series 
derived from the matching of the subject's calls against 
the reverse order of the pack). 


Recording Techniques for Individual Tests 

: Prior to the fall of 1938, all recording in the 
individual experiments was done by the subject under the 
careful surveillance of the experimenter. At the con- 


clusion of the recording of guesses for each run, the 
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subject brought his record to the experimenter, seated 
himself at her left, and recorded the card symbols as 
the experimenter called them from the target pack which 
lay face up on the table before her. Only after the re- 
cording was completed and this recording checked by the 
experimenter was the card removed (and turned). A cor- 
respondence or ‘hit' was indicated (by a horizontal line 
through call and card symbols) immediately upon being 
encountered and this hit was checked by both subject and 
experimenter against the card which lay face up on the 
table. At the end of each recording, the hits for the 
entire run were summed and the total score entered on the 
record sheets. It would thus appear that a genuine hit 
would rarely fail to be recorded except in the event of 
a miscall of the card by the experimenter. On the other 
hand, a recorded hit which was not genuine would require 
that the experimenter not only fail to check a particular 
recording but also fail to observe and check the record- 
ing of the hit line. This double failure is conceivable, 
To the authors, however, it would seem to constitute ex- 
treme experimental negligence.1 


That errors occur despite precautions is evident 
from the fact that in 1000 runs of the UT series (p. 175) 
which have been subjected to an intensive error search, 
there occurred 58 runs in which 64 recording errors were 
caught and corrected at the time of recording (these were 
primarily of the slip of tongue or of pen variety) and 5 
runs which were discarded due to violation of the 55555 
check. Of the 64 detected errors, 4 would have affected 
the score favorably, 5 adversely, the remainder leaving 
it unaffected, thus indicating that the preponderance of 
error is confined to the less-interesting and probably 
less attended non-hit regions. This suggests a slight 
relaxation of vigilance on the part of both experimenter 
and subject in these more monotonous stretches. That 
such 'relaxation' might have led to increased scores in 
the form of unmarked hits is, of course, possible. Ac- 
tually, 2 such hits were found, both in recordings in 
which the 55555 check was violated. 


Inversion errors induced by the subject's call 
series would appear to be excluded by the technique of 
recording since these errors require the retarded record- 
ing of a pair or sequence which has been called as a pair 





1. One subject, (in the 1937 series) assuming such negligence, made 
repeated attempts to fraudulently achieve high scores. Failing 
this, the subject discovered 'time pressure’, and withdrew from 
the experiment. The records were discarded. 

2. This check has, as a matter of course, been applied to all data. 
The 55555 check is a determination of the frequency of each syubol 
in the deck. There should be, in this case, five of each. 
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or sequence. The remote chance that there occur conse- 
cutive recording errors equivalent to the inversion er- 
ror and thus immune to the 55555 check is present but 
unimpressive as a scoring factor. The hypothesis that 
error of inversion might play an important role in extra- 
chance scoring has been entertained by the authors since 
the beginning and was, perhaps, decisive in the choice 

of a recording technique. 


Since September 1938, the recording has been 
done by the experimenter, the subject's guesses being 
shielded during the process and the 55555 check heing 
applied upon the conclusion of the recording of each run. 
The incidence of recording error as evidenced by detected 
error has shown marked increase under this procedure, 176 
instances of misrecording of some form having been noted 
and corrected during the experimental period. No errors 
have been found through application of the 55555 check 
subsequent to the experimental period. Since the added 
vigilance at the hit position was here missing, it would 
be anticipated that the incidence of error affecting the 
score would display an increase. Thirty-three such in- 
stances, 12 favorable to increased scoring, 21 unfavor- 
able, occurred. 


Test Supervision 


All experimental work was conducted under the 
supervision of at least one of the authors. This does 
not, however, imply that one of the authors was present 
during the administration of every individual test. 
Several student assistants participated in the test ad- 
ministration. These assistants, though not of graduate 
standing, were mature students whose major interests were 
in the fields of mathematics or of psychology. Integrity 
and intelligence, together with demonstrated reliability 
and established accuracy in performance, formed the basis 
for their selection. Compensation, supplied by grants 
from the National Youth Administration, was in no wise 
dependent upon the continuance or upon the outcome of the 
experiments. These student assistants were carefully 
trained, first as witnesses, later as experimenters under 
observation, before being entrusted with the independent 
administration of tests. They were assigned only to ex- 
ploratory tests or to the continued testing of subjects 
who had long maintained a relatively consistent scoring 
record. All test administration was subject to super- 
Vision in the form of witness or taking over by author 
D.M. without warning. No assistant was himself able to 
achieve extra-chance scores either in standard screened 
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tests or in such ‘off-the-record' tests as were self ad- 
ministered. No evidence in support of the hypothesis 
that scoring success is 3 function of the individual ex- 
perimenter was obtained. “® 


Procedure for Group Tests 


In the group tests, which were purely exploratory 
in nature, the following procedure was observed. Each 
member of the group was provided with a standard record 
sheet and was asked to record his guesses for the DT 
order of the pack which had been shuffled, cut, and 
placed by the experimenter on the deck behind a reading 
stand which served as screen both for the group and for 
herself. When all had finished recording guesses, the 
order of the target pack was slowly called to the sub- 
jects and recorded by them. The experimenter herself 
kept an independert record of the card order. Fach sub- 
ject was asked to indicate hits as noted and to sum his 
score before proceding to the next test. When, for pur- 
poses of completeness, it seemed imperative to include 
these tests in this report, the record of the experiment- 
er was compared with the guessed order of each subject 
and the official score (and the incidence of recording 
error on the part of the subject) noted. It is, of 
course, at once obvious that under this procedure an in- 
pressive scoring record could be fraudulently achieved 
by any individual subject through the simple device of 
deferred recording of guesses. This flaw in the proced- 
ure was exposed to each group at the end of the experi- 
mental period, and the reaction of each was such as to 
suggest that the thought had failed to occur to the men- 
bers. The essentially chance nature of the results would 
tend to indicate that this form of potential fraud was 
negligible in incidence, if extant at all. 





Official scoring of this series of 2840 runs re- 
veals in the unofficial student recording 607 instances 
of recording error, of which 149 affected the score posi- 
tively and 86 affected the score negatively. In Table 
0.1 these errors have been tabulated according to type. 
From this tabulation it appears that greatest error in- 
cidence results from the ‘chain error' in which one mis- 
recording in the run causes displacement of all succeed- 
ing card recordings. This type of error would, of course, 
be subject to immediate detection in individual tests. 
The relatively high incidence of miscellaneous error 





2a. Table 0.1A (Appendix) presents scoring averages obtained by each 
of several experimenters in a series of 1000 DT runs performed 
by a single subject. 
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which would have affected the score positively, suggests 
either deliberate misrecording or a tendency to record 
the symbol seen rather than the symbol called by the ex- 
perimenter.. The low incidence for the error of inver- 
sion is of peculiar interest to the authors since only 
this type of error or its equivalent could escape detec- 
tion on application of the 55555 check. Scoring errors 
which do not involve errors in recording are of little 
significance in an error study since they would, of 
course, be eliminated in any recheck of the data. 


Since recording conditions which obtained for 
these tests were as conducive to error as any which are 
conceivable, it would seem that the error data here pre- 
sented might offer something of an upper bound or, more 
conservatively, at least an average measure for such er- 
ror incidence as might be induced by the subject's un- 
shielded calls. 


CONTROL SERIES: THEORETICAL AND EMPIRICAL 


Theoretical Series 


Perhaps the most interesting controls are those 
theoretical ones deriving from the hypothesis that chance 
alone is operative in scoring success. On the assumption 
that the subject's calls are truly random--i.e., that 
each call is independent of all others--the distribution 
of scores is given by the binomial (4/5 + 1/5)*®, with 
mean = 5, and variance = 4. This distribution for 1000 
runs is given in Table 0.2. If, however, it be assumed 
that knowledge of the pack composition will constrain the 
subject to call exactly five calls of each symbol--a sit- 
uation obviously realized in the matching of two ESP 
packs against each other--the frequency distribution be- 
comes that of the matching hypothesis. Exact frequencies 
for this case have been supplied by Greville (4). For 
purposes of ready comparison with the frequencies of the 
binomial hypothesis, these frequencies for 1000 runs are 
included in Table 0.2. Greenwood (3) has shown that the 
variance for this case = 25/6. That either of these hy- 
potheses will obtain in practice is, of course, unlikely. 
Regardless, however, of the nature of the call series, it 
is established: 1. that the expected score per run is al- 
ways 5; 2. that the maximum possible value for the vari- 
ande (25/6) is that of the matching hypothesis; and §. 
that either the binomial or the matching hypothesis is 
adequate in supplying theoretical values for the distri- 
bution of successes (2, 3). It is of interest to note 
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that Greenwood (2), in an extensive empirical investiga- 
tion of a free calling chance series, has found the bi- 
nomial hypothesis to offer the es description of the 
data actually assembled by hin. 


Matching Series 


In order to test empirically the adequacy of the 
mathematical predictions, several empirical controls were 
introduced. The first was derived from 1000 matchings 
of two well-shuffled packs. For purposes of checking and 
analysis, these pack sequences were recorded. Results ar 
are presented in Table 0.2. This series will be termed 
Matching Series lI. 








A second matching series of 1000 runs was derived 
from the recorded pack sequences arising in a previously 
reported study (9). In this series, each recorded pack 
sequence was compared with the pack sequence consecutive 
to it and the coincidences noted, thereby subjecting 
each target pack to a mechanical or 'robot' calling. In 
order to achieve 1000 comparisons, the last sequence was 
compared with the first. This series, which is also pre- 
sented in Table 0.2, will be termed Matching Series II. 
This series supplies not only an empirical control on 
theoretical frequencies, but also a check on the adequacy 
of shuffling for the 1000 runs involved. 





Reverse Series 


In order to provide a more adequate empirical 
control, each run of the entire experimental series has 
been subjected to an additional and, we believe, valuable 
check. In it, the subject's guesses for each run were 
compared with the reverse (in general, the up through) 
order of the pack and the coincidences noted. The series 
thus resulting will be termed the reverse series. This 
series supplies a control which for each subject exhibits 
the same symbol preferences and pattern idiosyncracies as 
does the direct experimental series. Should the scoring 
effects of these preferences or pattern idiosyncracies 
be negligible (as theoretical considerations would in- 
dicate) this series should yield an essentially chance 
distribution. This it does (subject to such variation 
as may be introduced through extra~chance scoring on card 
13, the middle card, which is common to the two series). 








3, The binomial hypothesis will accordingly, with one exception, be 
used for theoretical predictions throughout this paper. In all 
references to critical ratio, however, the more conservative 
values derived from the matching hypothesis will be recorded. 
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I. PRELIMINARY SERIES 


During the fall of 1936 and the spring of 1937, 
exploratory experiments in the field of Extra-Sensory 
Perception were initiated. The procedure employed in 
this series was that of the DT or down through technique 
described in a previous section, with the exception that 
in this instance the tests were unscreened. Thirty-nine 
subjects participated in these experiments. All were 
university students; most of them were sophomores taking 
the required course in General Psychology. The group 
was composed of individual volunteers. They were almost 
equally divided as to sex. With one subject, Miss 
E.W.*, the G.E.S.P. technique’? was sybstituted. With 
another subject, Miss D.W., distance® was introduced as 
an experimental factor. With this subject 110 additional 
runs performed under distance conditions have been added 
to those already reported (7). 


For convenience of reference, those subjects 
who completed 100 or more runs will be termed major sub- 
jects; those who completed less than 100 runs will be 
termed minor subjects. Since a detailed report of re- 
sults achieved by each subject is already available (7), 
the rose) te gathered from minor subjects will here be 
grouped. 


Results of Preliminary Series 


A summary of results (both direct and reverse) 
for this preliminary series is recorded in Table 1.1. 
It will be seen that the direct series yields an aver- 
age of 5.80 hits per 25, with the highly significant 
critical ratio of 24.09, whereas the reverse series 
yields an average of 4.98 hits per 25 and the clearly 
non-significant critical ratio of -.40. 





4. A detailed account of the procedure with this subject is given 
in a previous article (7). 

5. A technique in which either clairvoyance or telepathy or both 
may be operative. 

6. A total of 200 were performed under ‘distance’ conditions. These 
include 60 runs (Av. 7.60, C.R. 9.87) im which the subject and 
experimenter were separated as far as a room 20'x20' would per 
mit; and 140 runs (Av. 7.84, C.R. 16.49) in which subject and 
experimenter were in adjoining rooms. 

7. For readiness of reference, scoring summaries for each of these 
minor subjects are included in Table 1.1A (Appendix). 
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es, 


Frequency data for the two series are recorded 
in Table 1.2 and are given graphic representation in Fig, 
1.2. For purposes of comparison, frequency predictions 
for the entire series based upon the binomial hypothesis 
are included in both table and figure. It will be ob- 
served that modes for both direct and reverse distribu- 
tions occur, in accordance with chance expectation, at 
5. It is to be noted, however, that the reverse fre- 
quencies correspond closely throughout the entire range 
with those predicted by chance (x2 = 8.58, n = 12, P = 
-75), whereas the direct series exhibits a significant 
excess of frequencies above 6 and a corresponding defici- 
ency of frequencies below 6 (x2 = 1673.10, n = 12, P = 
<107°°). This discrepancy between chance prediction and 
the direct series must be largely attributed to the per- 
formance of the six major subjects who achieved average 
scores which on the basis of a 2.5 critical ratio cri- 
terion®, may be regarded as statistically significant, 
Individual frequency distributions for these subjects re 
veal that for Miss E.W., the mode occurs at 8; for Miss 
D.W., for Mr. C.J., and for Miss B.C., at 7; for Mr. C, 
B.. at 6; and for Mr. E.A. (whose average score of 5.60 
with C.R. of 3.94 is not impressive), it appears at 4, 
It is interesting to note, however, that it is this lat- 
ter subject who has scored the highest number of hits 
per run ever to be obtained in this laboratory. For 
readiness of comparison, an aggregate distribution table 
for ne high-scoring subjects and for all other sub- 
jects’ has been presented in Table 1.2. The contrast be- 
tween the performance of those subjects who score signi- 
ficantly and those who do not becomes increasingly evi- 
dent on further detailed analysis. The records of all 
subjects have, accordingly, for this and for all subse- 
quent series, been subjected to analysis with respect to: 
1. daily fluctuation in scoring, 2. frequency of conse- 


cutive hits, and 3. frequency of hits with respect to 
card position. 


Daily scorin averages for each subject in this 
series appear in Table 1.3A+%. It is of interest to note 
that on July 27, Miss D.W. for the first time approached 


8. For purposes of classification, this basis for significance has 
been arbitrarily adopted throughout this paper. The authors 
should, however, hesitate to accept this criterion as an adequate 
basis for crucial decision. 

9. Of these low-scoring subjects only one achieved a C.R. below -2.) 
This subject in 30 runs attained an average score of 4.00 with 4 
C.R. of -2.60.. 

10. Tables which bear the label 'A' appear in the Appendix. 
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a chance level of scoring. On inquiry, it was learned 
that she had neglected lunch and had devoted the two 
previous days---except for a few hours stolen for sleep-- 
to the typing of a thesis- Paradoxically, she here dis- 
played for the first time, interest in her scoring abil- 
ity, expressing confidence that when rested she could 
again achieve her previous scoring record. Her accept- 
ance of a teaching position and departure shortly there- 
after from the university made it impossible to put this 
to the test.. 


In Table 1.4 is presented the tabulation of con- 
secutive hits (direct and reverse) for both major and 
minor subjects together with the predictions (on the 
basis of the binomial hypothesis) for these consecutive 
hits. The table reveals consistent differences both be- 
tween the direct and the reverse series and between the 
direct series and chance predictions as against a close 
parallelism between the latter and the reverse series. 
From this table it would appear that for those subjects 
who score significantly, the discrepancy between direct 
and reverse series results not only from an increased 
frequency of isolated and paired hits, but also from the 
greater frequency of consecutives of higher order. 


Summaries of hit frequency with respect to card 
position for both direct and reverse series are present- 
ed in Table 1.5A. This table contains a tabulation of 
the frequency of scoring for the top card, the second 
card, the third, etc. In Table 1.5, average scores (per 
25) for the first, second, and succeeding five-card 
groups are recorded. These analyses, undertaken as a re- 
sult of experiments previously published and later to be 
reviewed in this paper (p. 29), reveal for this series 
a slightly W-shaped pattern for scoring success. This 
would seem to confirm findings of Rhine, Pegram and 
others (11, 10, 1) who report for the DT method a Wor 
U-shaped curve of operation. Comparison with the reverse 
data, together with statistical treatment (maximum criti- 
cal ratio of the difference between direci scores for the 
j-card groups being 2.02) fails, however, to establish 
for our aggregate results clear statistical Significance, 


When, however, records for high-scoring and for 
low-scoring subjects are isolated, it is noted that a 
somewhat more pronounced W-shaped curve obtains for the 
former (with maximum critical ratio of the difference 
equal to 2.47), whereas for the latter a U-shaped curve 
is obtained (with maximum critical ratio of the difference 
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equal to 1.57) thus suggesting a possible association 
between the W-shaped curve and above chance scoring suc- 
cess. Examination of records for individual subjects 
tends somewhat to confirm this position. For only two 
subjects, however, are the scoring trends sufficiently 
pronounced to command statistical support. For one of 
these subjects, Miss E.W., the W-shaped curve fails to 
obtain, and scoring success appears to vary directly with 
the distance of the card from the top of the pack. This 
might seem impressive were it not for the fact that in 
this single instance the standard DT procedure was re- 
placed by the G.E.S.P. procedure. Again it would appear 
that a clearly significant relationship between scoring 
success and card position fails to emerge. 





More striking, however, in this connection, is 
the rather consistent incidence of high scoring on the 
first and last cards of the pack. This at once suggests 
that, despite experimental precautions, adequate protec- 
tion against sensory cues may not have been given these 
cards. In many instances, the subject gave no attention 
to the shuffling process or sat with back toward the ex- 
perimenter, thus rendering inapplicable the hypothesis of 
sensory cues. In the case of Miss D.W., who exhibits in 
most marked degree this tendency toward high scoring on 
cards 1 and 25, it will be recalled that 200 runs were 
performed under ‘distance’ conditions. These runs yield 
77 hits (with an average of 9.64 per 25) for the first 
card, and 107 hits (with an average of 13.33 per 25) for 
the last card of the pack. There are instances, however, 
in which the hypothesis of sensory cues might apply. For 
this reason, adjustment has been made for scoring on the 
first and last cards of the pack, by assigning to these 
cards a score derived from the average scoring on the ad- 
jacent four cards. These corrections are presented par- 
enthetically in Table 1.5. It is noteworthy that this 
adjustment does not essentially alter the scoring pattem. 
It does, however, reduce the statistical significance 
which may be assigned to the observed variations. That 
the superiority of direct over reverse scores for high- 
scoring subjects does not derive from this high scoring 
on the first and last card positions is at once obvious. 
It is, moreover, to be noted, (Table 1.5A) that for each 
of these subjects (with the single exception of Mr. E.A:), 
direct scores exceed chance expectancy for all card posi- 
tions. 


II. GROUP SERIES 


The group tests, which were quite exploratory in 
design, were conducted primarily as class room 








~~ wae tones -_ 





for 


ver, 
For 
the 
se 
ad- 
ar- 


tern, 


h- 
ng 
us. 
ach 
A.), 
osi- 





STUDIES IN EXTRA-SENSORY PERCEPTION 171 





demonstrations. Two hundred and «ighty-four students 
participated. The procedure for these tests has been de- 
scribed in detail above (p.164). It will be recalled 
that it was such as to admit these tests to the screened 
category. Ten runs were performed by each subject. 


Results of Group Tests 


In Table 2.1 is presented a summary of the re- 
sults both direct and reverse for this group series. The 
frequency distribution for the entire series appears in 
Table 2.2. This distribution is given graphic represent- 
ation in Fig. 2.2. Summaries for daily fluctuations in 
scoring, for consecutive and for card position frequen- 
cies are contained in Tables 2.3A, 2.4, 2.5 and 2.5A. A 
distribution of total scores attained by individual sub- 
jects is contained in Table 2.6 and is presented graphi- 
cally in Fig. 2.6. 


From Table 2.1 it appears that the results here 
obtained (average score = 4.995, with C.R. = -.12) are 
in rather perfect accord with chance expectation. That 
this chance character of the results obtains not only 
with respect to score, but also with respect to frequen- 
cy distribution and to consecutive and card position fre- 
uencies, is evident from Tables 2.2, 2.4, and 2.5. From 
table 2.6 it appears that no subject achieved notable 
scoring success, 


Of interest is the character of the distribution 
of individual totals in Fig. 2.6. Whether additional 
trials (or increase in the number of subjects tested) 
would have accentuated the apparent trimodality of the 
direct distribution, thus suggesting the presence of in- 
dividuals from three diverse populations, is a matter for 
speculation. The authors, having discovered no subjects 
who consistently maintain scores below the chance level, 
should anticipate the disappearance of at least one of 
these apparent ‘modes’. 


III. MAIN EXPERIMENTAL SERIES 


The Main Experimental Series was initiated in the 
fall of 1937. All tests were performed under screened 
DT conditions described in the above section on General 
Procedure for Individual Subjects. 





The subjects participating in this series will 
again be classed as major or minor, the term major being 
reserved for those who completed the 500 runs set as an 
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arbitrary minimum for this series. The minor subjects 
were those who were unable to allow sufficient time for 
this arbitrary minimum. The major subjects were, with op 
exception, interested students, most of whom had read 
Harper's account of the work at Duke University (12), ang 
who had something of a bias for or against the ESP hypo- 
thesis. The group tests administered in the classes wer 
singularly unfruitful in supplyjng subjects who were read 
to submit to sustained testings* Tests with each subject 
in this series were continued as long as his convenience. 
and interest--permitted. In no case was a subject dropps 
becaus¢ of low scoring or pressed to go on if he scored 
high. The number of runs performed during a given work 
period were determined by the time available and by the 
subject's tempo. The minimum number of runs for any ex- 
perimental period was ten. 


Some comment with respect to major subjects may 
be cf interest. Mr. C.B. was an upper class student with 
a major in psychology. His attitude was that of the in- 
terested experimentalist, curious and non-committal. Mr, 
H.D., a sophomore with a flair for the unusual, was fas- 
cinated with the idea of the experiment, and offered hin 
self as a subject with a rather certain feeling of suc- 
cess. Despite his unimpressive scoring, he retains both 
interest and 'faith'. Mr. R.S., a sophomore premedic 
student, with a wide range of interests and of abilities, 
proposed himself as a subject in order that he might as- 
sist in ‘blowing the hypothesis to bits'. He tried vari- 
ous guessing techniques which he hoped would result in er 
tra-chance scores, but failed to achieve his objective, 
It was then suggested by the experimenter that he suspend 
his rational attack in favor of recording the first syn- 
bol that occurred to him. To his intense amazement, he 
began immediately to achieve extra-chance scores. He 
persisted with the firm conviction that his 'run of luck’ 
would break. His present attitude is one of puzzled ac- 
ceptance of extra-chance scoring as a phenomenon which 
demands further study. Miss D.M. is one of the au- 
thors. 43 She submits to the tests only when she is too 





11. They did, however, contribute several minor subjects. 

12. It must be noted, however, that high scoring is conducive to 
maintenance of interest in the experiment and that this factor 
of interest often has direct bearing upon the willingness of 
the subject to submit to sustained experimentation. In this 
sense, individual experiments tend to be weighted somewhat in 
favor of the high-scoring subject. 

13. Miss F.P.S., the other author, had performed numerous sel f- 

administered tests prior to the initiation of our experimental 

work. No records of these tests were kept. Scoring was some- 
what above chance. 


| 
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weary for other exertion. Mr. C.J., our subject extra- 
ordinary, who has continued experimental work throughout 
his last three years of college, presented himself as a 
subject, eager to try the tests, and confident that he 
would be able to score above chance. This confidence 
seemed to spring from several ‘notable coincidences' in 
his own experience and from his trust in his mother's re- 
liable ‘intuitions'. During the first two years of ex- 
perimentation, he was not only definitely interested in 
the ESP hypothesis, but seemed to derive genuine satis- 
faction from the actual testing procedure. During the 
past year, however, the fun has apparently vanished from 
the actual process of card guessing, and he has continued 
primarily because of the sense of responsibility which he 
holds toward the work. This very natural ennui has been 
accentuated by the nervous tension and fatigue resulting 
from excessively heavy academic and activity programs. A 
slower tempo of performance has resulted. In consequence 
of this slower tempo and of the subject's limited avail- 
ability, it has been impossible to achieve the completion 
of the entire schedule of experiments which had been 
planned for the year. 


Results of Main DT Series 


In Table 3.1 is presented a detailed summary of 
the results (both direct and reverse) for this series. 
It will be seen that the entire series of 4180 runs 
yields a direct average of 5.79 hits per 25, with the 
highly significant critical ratio of 25.19, whereas the 
reverse average is 4.97 hits per 25, with the clearly 
non-significant critical ratio of -.90. 


Examination of individual results for both major 
and minor subjects reveals that these subjects present 
marked variation in scoring success. For the high-scor- 
ing major subjects, Messrs. C.J., R.S., and C.B., the 
average scores of 6.89, 6.03, 5.50, and the critical 
ratios 29.35, 11.25, and 6.52, respectively, are such as 
to indicate either the occurrence of several extremely 
unusual chance events, or high probability that the re- 
sults are extra-chance. On the other hand, scores of 
5.17, and 5.06, with critical ratios of 1.84, and .66, 
for Miss D.M., and for Mr. H.D., respectively, reveal 
the essentially chance character of these records. Sim- 
ilar variations in scoring success are exhibited by the 
minor subjects, of whom four obtain scoring averages sig- 
nificantly above chance. For readiness of comparison, a 
Summary for high and for low-scoring subjects is present- 
ed in Table 3.1. This contrast between subjects becomes 
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even more striking when their records are subjected to 
the further customary analysis with respect to distribu- 
tion of hits, frequency of consecutive hits, daily fluc- 
tuation in scoring, and frequency of hits with respect 
to card position. 


In Table 3.2 appear the frequency distributions 
(direct and reverse) for both major and minor subjects, 
Frequency distribution summaries for high and for low 
scoring subjects are presented in Table 3.2 and are given 
graphic representation in Fig. 3.2. The marked contrast 
between the two groups of subjects is again evident, both 
with respect to the location of the mode and with respect 
to the frequency of occurrence of scores above the mode, 


Daily scoring averages are presented for each sub- 
ject in Table 3.3A. Of primary interest is the consis- 
tency with which above-chance scoring is maintained by 
the majority of high scoring subjects. Noteworthy, how- 
ever, is the prolonged decline and eventual recovery ex- 
hibited by the scoring of Mr. C.B. With Mr. C.J., on the 
other hand, it is interesting to note that on October 
26th, and on that date only, scoring dropped to the chance 
level. This drop in scoring level occurred on the Monday 
following a vigorously celebrated Homecoming week-end at 
the University. 


In Table 3.4 is presented a tabulation giving for 
each subject the frequencies for isolated and for conse- 
cutive hits, together with the predictions on the basis 
of the binomial hypothesis, for these frequencies. Again 
it appears that the advantage enjoyed by high-scoring 
subjects resides primarily in their achievement of conse- 
cutive hits of the second and of higher orders. 


From Mr. C.J.'s records for this series it became 
at once apparent that for him the level of scoring 
throughout the pack was not constant, higher scoring suc- 
cess being achieved for those cards which were situated 
near the top of the pack. It was this observation which 
prompted the inclusion in all series of the analysis of 
scoring success with respect to card position. This an- 
alysis for all subjects in this series is presented in 
Tables 3.5 and 3.5A and is given graphical representation 
in Fig. 3.5. It may be noted that the aggregate data 
catabtial highest scoring for the first five cards in the 
pack, and lowest scoring for the last group of five, the 
maximum critical ratio of the difference between scores 
for successive 5-card groups being 4.34. When, however, 
the performance of various individual subjects is scrut- 
inized, it is noted that wide variation occurs. (For no 
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subject does the W-shaped curve found in the preliminary 
series appear.) Since the scoring variations exhibited 
by Mr. C.J. are so pronounced as to dominate the entire 
series, it has seemed desirable to present summaries for 
the high-scoring subjects and for the entire series both 
with and without the data for him. These summaries are 
included in Tables 3.5 and 3.5A. From these it appears 
that for both major and minor high-scoring groups, ex- 
clusive of Mr. C.J., maximum scoring success is achieved 
in the fourth 5-card position and minimum in the last 5- 
card position. Statistical treatment, together with con- 
siderations of consistency, would, however, indicate that 
only the latter variation may be regarded as significant. 
For the low-scoring subjects, variations in scoring are 
clearly non-significant. 


IV. EXPERIMENTAL VARIATIONS WITH MR. C.J. 
A. The UT Series 


In consequence of the findings in the Main Experi- 
mental Series, there appeared a definite need for further 
intensive study of individual subjects. Extensive ex- 
perimental variations were, accordingly, undertaken with 
the subject who furtunately combined high scoring ability 
with availability---Mr. C.J. 


In the attempt to discover whether scoring abil- 
ity might be subject to voluntary change in direction, 
and to further investigate the remarkable relationship 
previously found to obtain for this subject between suc- 
cess in scoring and the position of the card in the pack, 
& second series of 1000 runs was initiated in January of 
1938. This series will be termed the UT series. 


The experimental procedure was identical with 
that reported for the Main Experimental Series with the 
Single exception that the subject was here invited to 
guess the up through rather than the down through order 
of the pack. It was in this series that the 'robot' 
check (Watching Sertes II) was introduced as an additional 
control. It becomes obvious that the control which has 
been called the reverse series would here be obtained by 
comparing the subject's guessed order (intended for the 
up through pack order) with the unintended or down 
through order of the pack. 


Results for the UT Series 


This series, together with an analysis of the 
data obtained, has been published in a previous repert. 
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report. (9). For a detailed discussion of these results, 
the reader is, accordingly, referred to this report. 


A summary of results for the series, presented ip 
Table 4.1, reveals a direct average of 7.39 with the high- 
ly significant critical ratio of 37.01, in marked contrast 
to reverse and matching averages of 5.13 and 4.91, re- 
spectively, with non-significant critical ratios of 1.95 
and -1.38, respectively. 


Hit frequencies for this series appear in Table 
4.2; daily fluctuations in scoring in Table 4.3A; conse- 
cutive hit frequencies in Table 4.4. Card position data 
are contained in Tables 4.5 and 4.5A and are given 
graphic representation in Fig. 4.5. 


From Table 4.5A it appears that the striking re- 
lationship between success in scoring and card position 
found to obtain for this subject in the DT series, is 
here maintained despite the reversal in technique. Ap- 
plication of the Spearman Rank Difference Method of cor- 
relation here yields = .94 + .02. The reverse and 
matching series, on the other hand, reveal an apparently 
random distribution of hit frequencies with respect to 
card position, p = .30 + .13, and p = .07 + .14, respect- 
ively. That this observed relationship between scoring 
success and card position is more than a statistical arti- 
fact is a postulate which appears worthy of further ex- 
amination. 


B. The EE Series 


Series EEp; and EE; p 





In order to further test the apparent relation- 
ship between scoring success and card position, addition- 
al experimental variations were introduced. Primary among 
these variations is that of the EE (equal exposure) ser- 
ies which was undertaken in the fall of 1938. In this 
series, a screen 57" long and 22" high was so placed be- 
tween the experimenter and the subject as to shield con- 
pletely both cards and experimenter from the vision of 
the subject. The experimenter shuffled and cut the pack 
and then dealt out all 25 cards face down in a single 
row parallel to the screen. The subject then recorded 
his guesses for the cards from right to left or from left 
to right as designated by the experimenter. As previous- 
ly noted (p. 6) all card recording was done by the ex- 
perimenter (and witnessed by the subject), the subject's 
calls being shielded during the recording process. In 








res 
sle 
thi 


for 
spe 
DT 
for 
EE 


: 
E 
a" 
res 
ter 
of 
and 
vot 


for 
for 
day 
nar 
Tab 


Ser 


ser 
sma 
to 

fro 
rig 
pac 
tim 
ies 


ed 
tio 
pea 


Ser 


twe 
tio: 











STUDIES IN EXTRA-SENSORY PERCEPTION 177 





response to the request of the subject, who chafed at the 
slower tempo (resulting from inter-run delay) imposed by 
this experimental procedure, and who was eager to re- 
establish his previous scoring record, 140 DT runs (per- 
formed under the same screened conditions) were inter- 
spersed among the first 140 runs of the EE series. These 
DT runs, by establishing the subject's DT level of scoring 
for this particular period, offer a needed control on the 
EE series. 


The EE series includes two principal sub-series. 
The first, EE | consists of 400 runs in which the order 
of the tet ee s calls was from right to left; the second, 
EE, Rp, consists of 140 runs in which the reverse or left 
to right order obtained. When it became evident that the 
results of this second series exhibited not only high in- 
ternal consistency but also close parallelism with those 
of the RL series, the series was arbitrarily terminated, 
and the short time available for experimentation was de- 
voted to additional experimental variation. 


In Table 5.1 is presented a summary of results 
for these series (EEpy,, EEL p; DT). The distributions 
for hit frequencies appear in Table 5.2 and Fig. 5.2; 
day-by-day averages for this period in Table 5.3A; sum- 
maries for consecutive hits and for card position data in 
Tables 5.4, 5.5 and 5.5A. 


Series EE O5-5 


An examination of the results of the two EE 
series prompted the division of the subject's task into 
small experimental units. It was, accordingly, decided 
to introduce EE 5- in which only five cards were dealt 
from the pack at a time; the subject's guesses for the 
right-left order of each set of five (within the given 
pack) were then recorded. Scoring was deferred until 
guesses for the entire pack had been recorded. Owing to 


ye pressure, only 110 runs were performed in this ser- 
es, 


The summary of results for this series is includ- 
ed in Table 5.1. Frequency distributions, daily fluctua- 
tions, consecutive hit and card position frequencies ap- 
pear in Tables 5.2, 5.3A, 5.4, 5.5 and 5.5A, respectively. 


Series EE5 


In order to create a more pronounced break be- 
tween consecutive sets of five, a second and last varia- 
tion was introduced in the form of EE;. The procedure 








178 JOURNAL OF PARAPSYCHOLOGY 





was here identical with that of EE 5~ except that here 
the cards were returned to the oact abd the entire pack 
was again shuffled before the succeeding set of five 
cards was dealt. Recording and shuffling necessarily 
occurred at the end of each set of five calls. Since 
the recording was here subject neither to witness nor to 
the 55555 check, each recorded set of five was immediate- 
ly subjected by the experimenter to a second check against 
the target cards. For convenience of reference, a set of 
five consecutive runs is treated as a run of 25. Sixty 
such runs were performed. Results and customary anal- 
en are included in Tables 5.1, 5.2, 5.34, 5.4, 5.5 and 
5A. 


Discussion of Results for the EE Series 


The Summary Table 5.1 reveals that for these 
series, the subject's scoring level (average 6.16 and C.R, 
of 16.58) is, in all but the short EE; series, consistent- 
ly below previous levels attained by Rim. That this low 
ering of the scoring level can scarcely be attributed to 
the shift to the EE technique seems apparent from inspec- 
tion of the control DT series. In fact, the average 
score (6.23, with C.R. = 16.09) for EE exceeds that for 
DT (average 5.79 with C.R. = 4.60). While statistical 
treatment (C.R. diff. = 2.34) suggests that this differ- 
ence may be significant, the authors feel that comparison 
between the two series should be confined to that data 
accumulated over the same experimental periods. When ap- 
plied to the DT series and to those runs (also 140) of 
the EE series which were performed during the correspond- 
ing experimental periods, statistical treatment yields a 
critical ratio of 3.33 for the difference between the 
mean scores of 5.79 for the former and 6.60 for the lat- 
ter. That the EE technique may be more conducive to 
extra-chance scoring than is the DT technique is indi- 
cated by these results. Similar observations to the ef- 
fect that lower scoring seems associated with the DT 
technique have been recorded by others (1, 11). Again 
a possible relationship between scoring success and a 
physical factor (card exposure) is suggested. This 
postulated superiority of the EE technique as a basis 
for scoring success gains support if attention is direct- 
ed to the scoring level maintained in the EEp; series 
for the first five cards of the pack. During 110 conse- 
cutive runs of this series, there was obtained for these 
five cards an average score of 11.00 hits per 25---a 





14. The series was terminated by the subject's departure from school. 
He returned only for the commencement exercises. 
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scoring level never maintained during the subject's peri- 
ods of optimum scoring for a comparable number of con- 
secutive runs of a DT or UT series. Even if the entire 
BE series (which includes 100 runs of approximately 
chance scoring in January of 1939) be included, the scor- 
ing average of 8.39 established for the first five cards 
of the pack is comparable to the scoring averages ob- 
tained for these cards at the peak of the subject's scor- 
ing success. 


Worthy of note, moreover, is the average score 
of 13.17 maintained throughout the 60 runs of EEr- for 
the last 5 cards in each set of 25. This averagé has 
only once (and then for 20 runs) been equaled by this 
subject for any 5-card group and has never been ap- 
proached for the last 5 card positions in a DT or UT 
series (the maximum score previously recorded for these 
positions being one of 9.33 maintained for 30 consecu- 
tive runs). 


Inspection of Table 5.3A reveals for these 
series marked departure from the consistency in scoring 
which has previously characterized the performance of 
this subject. Notable, moreover, is the improvement 
both in the level and in the consistency of scoring sub- 
sequent to March 3, 1939. Whether this improvement re- 
presents a return by the subject to his customary 'form', 
or a response to variation in technique (i.e., to divi- 
sion of the task into smaller units) is a matter for 
speculation. In the absence of adequate data and con- 
trols, the authors incline to the position that both 
factors were operative. 


As would be anticipated in consequence of the 
reduced average scoring level, the excess of consecu- 
tive frequencies over predicted frequencies is, for these 
series, less marked than for previous ones. 


Examination of card position data for EE series 
(exclusive of EE-) reveal for these series high scoring 
for the first fite cards of the pack, near chance scor- 
ing for the intermediate cards, followed by varying de- 
* of scoring recovery for the last five cards. For 

ERL and for EE, p this 'recovery' fails to command sta- 
tistical ne (the maximum critical ratio of the dif- 
ference between scores for the last and for the inter- 
mediate 5-card positions being .82 and .13, respective- 
ly); for EEor_5, on the other hand, it receives strong 
statistical Retort (the maximum critical ratio of the 
difference being 8.95). Only in the arbitrary runs of 
25 formed from the EE; series, is the approximately 
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constant level of scoring which might have been antici- 
pated for the EE series maintained. The DT series, which 
here serves as a control, exhibits a slight but statis- 
tically non-significant U (or W)-shaped trend line. 


C. MINOR VARIATIONS 


After the completion of the long DT series and 
prior to the initiation of the UT series, several minor 
experimental variations with Mr. C.J. as subject were 
initiated. In these short series, it was our purpose to 
so accustom the subject to experimental variations that 
these would entail neither loss of confidence nor disrup- 
tion of scoring. Eventually it was hoped that variations 
unknown to the subject (e.g., blank cards) might be in- 
troduced. 


DT-Ten Packs (DTr) 


The first variation introduced was that of using 
ten different packs for each set of ten runs. This vari- 
ation serves the twofold purpose: 1. of providing a sit- 
uation free from the inadequate shuffling criticism (in 
which it has been maintained that unexpected scores 
might be achieved by the subject through the conscious 
or unconscious recall of sequences which might persist 
in consecutive shuffles), and 2. of providing a situa- 
tion which requires, on the part of the subject, the 
ability to focus his scoring efforts upon a particular 
pack. The series thus obtained will be termed DTr. 





Ten ESP packs were shuffled and placed face 
down in a row at right angles to the screen. Pack 1 was 
removed from its position, cut, and placed directly be- 
hind the center of the screen. The subject was then 
asked to record his guesses for the DT order of this 
particular pack. The pack was checked as usual; the 
subject recorded; the experimenter turned and called the 
cards and noted each recorded symbol and each hit as it 
was recorded. Pack 1 was then returned to its initial 
position, and pack 2 was moved to the central position. 
This practice was continued until all ten packs had been 
called. If time for more than ten runs was available, 
the subject left the room; the set-up was again prepared; 
and on the subject's return, the procedure was repeated. 
Since the ten target packs were still in position, a se- 
cond check on the recorded card sequences was made at 
the end of each set of ten runs. Subsequent to the ex- 
periment, calls were checked against the nine pack se- 
quences toward which they were not directed. 
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Results for TDq 


A summary of the results both direct and reverse 
for this series and for the control series derived from 
the nine pack check is presented in Table 6.1. Hit dis- 
tributions for these series appear in Table 6.2 and Fig. 
6.2. Tables 6.3A, 6.4, 6.5, and 6.5A present summaries 
for daily fluctuations, for consecutive and for card po- 
sition frequencies. A more detailed analysis of the sub- 
ject's success in focusing his scoring effort is offered 
in Table 6.6, which exhibits the number of hits obtained 
by matching of each call sequence against all packs on 
the table. In this Table it will be noted that the dia- 
gonal cells represent the ‘voluntary' hit frequencies; 
all other cells contain the ‘involuntary' or chance hit 
frequencies derived from the non-target packs. It is to 
be noted that the totals exclude the diagonal cell en- 
tries, and thus represent only the involuntary hit tot- 
als. Column totals exhibit all involuntary hits for 
call sequences 1, 2, 3, etc. 





The results clearly indicate that high scoring 
(average 8.17, with C.R. 16.31) results from the match- 
ing of the subject's calls against those packs for which 
the calls were intended, whereas no extra-chance aver- 
ages derive from the matching of the calls against those 
packs for which they were not intended. It would thus 
appear, contrary perhaps to expectation, that the scor- 
ing effort of this subject admits of a high degree of 
voluntary direction. 


Worthy of note again in this series is the con- 
sistency in scoring, the excess in consecutive hit fre- 
quencies, and the observed direct relation between scor- 
ing see and position of the card in the pack. (p = 
82+. . 


DTi-Delayed Knowledge of Score (DTp) 


A second variation was introduced in order to 
test the motivating strength of the subject's knowledge 
of his score. It was felt by the subject that know- 
ledge of the score on a given run exerted a positive 
motivating influence on the mpg eet | trial, that is 
to say, if the score were low, he would attempt to bet- 
ter it; if it were high, he would try to match it. Ac- 
cordingly, a short series in which the subject's know- 
ledge of his score was deferred until after the comple- 
tion of ten runs was introduced. The procedure was the 
customary DT procedure with the following variation in 
Tecording. On the completion of the subject's calls, a 
record of the pack sequence was made by the experimenter 
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on a separate record sheet. The subject's calls were 
then read to the experimenter who recorded them and 
checked the hits. Since the experimenter's recording 
of the pack sequence was here without witness, a re- 
check of the recorded order against the target pack was 
made by the experimenter before proceding to the next 
run. The subject's independent record of his own calls 
was available for a subsequent call check. 


Results of DTp 


A summary of results together with the customary 
analyses with respect to frequency distribution, scoring 
fluctuation, consecutive and card position frequencies 
appears in Tables 6.1, 6.2, 6.3A, 6.4, 6.5 and 6.54. 





The results of this series, average 6.50 and 
C.R. 8.09, indicate that despite deferred knowledge of 
score the subject was able to maintain a scoring average 
well above the chance level. Comparison of the average 
for this series with that of the immediately preceding 
DTy series would seem to suggest that deferred knowledge 
is associated with depression of the scoring level. Ex- 
amination of the records reveals, however, exaggerated 
daily fluctuations in scoring (average 4.80 for the 
first experimental period of 30 runs, average 7.80 for 
the following period of 50 runs, and average 6.17 for the 
last period of 40 runs) rather than a consistently de- 
pressed scoring level. Since, however, the subject's 
reaction to this first major variation in experimental 
technique may represent an uncontrolled factor in the 
first thirty runs of the series, and since the long holi- 
day intermission 5 (Dec. 1 to Jan. 5) may have a bear- 
ing upon scoring for the last experimental period, it 
is difficult to assign any clear significance to the ob- 
served depression in scoring. 


More striking, however, is the shift in scor- 
ing pattern exhibited by this series, the linear trend 
of the previous series (DT, DTy) being here replaced by 
a U (or W)-shaped trend line. While the elevation in 
scoring for the last 5-card position which produces this 
pattern shift does command a certain statistical support 
(the maximum critical ratio of the difference between the 
last and the intermediate 5-card groups being 2.69), this 
support is challenged by the reverse series which yields 
@ corresponding maximum critical ratio of 2.94. The need 
for extreme caution in the interpretation of results 





15. Depressed scoring immediately following long vacation periods 
seems rather typical for this subject. 
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based upon so small a number of runs is obviously indi- 
cated. Despite the ambiguity of these findings, it is 
however to be recalled that similar U (or W) curves ap- 
peared in the Preliminary (unscreened) series and in the 
1938-1939 DT series, and have been found by others (1, 
10, 11) to be associated with the DT technique. 


DT-Ten Packs--Delayed Knowledge (DTqp) 


When it became apparent that high scoring could 
be maintained despite delayed knowledge of score, the 
DTy series was quickly terminated in order that a com- 
bination of the ten pack (DT7) and the delayed knowledge 
(DTp) techniques might be effected. 





In this series, DTpp, as in the DTy series, the 
ten packs were shuffled col placed face down in a row at 
right angles to the screen. Pack 1 was removed from its 
position in the row, cut, and placed in the center posi- 
tion behind the screen. The subject then recorded his 
calls for the DT order of this pack. Upon receiving the 
subject's signal that he had completed the recording of 
his calls, the experimenter recorded the pack sequence on 
a separate record sheet. The subject then read his calls 
to the experimenter who recorded them and checked the 
hits. Pack 1 was then returned to its initial position 
and pack 2 placed in the center position and the experi- 
ment continued. As in the previous tests with ten packs, 
a second check of the recorded pack orders against the 
ten target packs was made by the experimenter at the 
conclusion of each set of ten runs. As before, the sub- 
ject's calls for each pack were available for a subse- 
quent call check and for checking against the nine non- 
target packs. 


Results of DT rp 


Results for this series are presented in Tables 
6.1, 6.2, 6.3A, 6.4, 6.5 and 6.54. 


That high scoring (average 7.30 with C.R. 9.43) 
should be maintained under this ten-pack-delayed-know- 
ledge technique was a source of genuine satisfaction to 
the subject. Of psychological interest is the fact 
that knowledge of results plays a relatively unimportant 
role in the scoring success of this subject. 





Card position frequencies again exhibit (though 
in less marked degree) the usual pattern of declining 
hit frequencies with position of the card in the pack. 
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Scoring on the non-target packs continues, as 
in DTy, to be that of chance expectancy (average 4.99 
with C.R. -.02). Here again is evidence that the sub- 
ject's scoring efforts can be focused with remarkable 
success on a particular target pack. 


UT Variations: UTp, UTs59, and UTvs. 

Subsequent to the completion of the UT series 
and during the few remaining days of the Spring quarter 
of 1938, three short series were undertaken in order to 
further test the relationship between scoring success 
and card position. 





Since the DTp series yielded data which failed 
to support the previously observed relationship between 
card position and scoring success, it seemed desirable 
that deferred knowledge again be introduced as an experi- 
mental factor. This was done in a short UT series-- 
UTp--of 90 runs. The procedure for this series was iden- 
tical with that for the DTp series except for the re- 
versal in the order of eaklise the pack. 





Results of UTp 


The results of this series are presented in 
Tables 6.1, 6.2, 6.3A, 6.4, 6.5 and 6.5A. It is to be 
noted: 1. that despite the excellent scoring here 
achieved, the average 6.74 is below that for the immedi- 
ately preceding 100 runs of the UT series (average 7.09); 
2. that there again occurs that shift in scoring pattern 
which was observed in DTp, and which is characterized 
by depression in scoring for the central cards of the 
pack and elevation for the bottom ones. While this pat- 
tern variation commands only ambiguous statistical sup- 
port (maximum critical ratio of the difference between 
scores for the last and for intermediate 5-card posi- 
tions being 2.89 both for direct and for reverse series), 
its sudden reappearance with the re-introduction of the 
deferred knowledge factor cannot be entirely ignored. 


As a further variation, it was determined pro- 
gressively to increase the size of the pack in order to 
ascertain whether the apparently typical direct relation- 
ship between hit frequency and card position would be 
maintained, and if so, whether a depth of pack could be 
obtained for which scoring on the bottom cards would be 
approximately reduced to a chance level. The target 
packs were accordingly enlarged to 50 and 75 card size by 
the pooling of two or of three standard packs. The 
series pegnes tet from the use of these packs will be 
designated as U 50 and UTy5, respectively. The former 
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comprises 15 runs (of 50 calls), the latter only three 
runs (of 75 calls). These series were to have been con- 
tinued in the fall of 1938. In view, however, of the 
very limited time which the subject could make available 
to us, variations which gave promise of more decisive re- 
sults were at that time introduced. These series, ac- 
cordingly, remain unfinished. 


Results for these series (UT50, UT 5) are pre- 
sented in Tables 6.1, 6.2, 6.3A, 6.45 and bs, For pur- 
poses of the distribution table, scores are given in 
terms of 25-card runs. 


While inadequacy of data renders conclusions un- 
warranted, it is of interest to note that a scoring trend 
not inconsistent with that established for the DT and UT 
series is here exhibited, highest scoring success being 
again achieved for the top cards of the pack and lowest 
for the bottom cards. 


SUMMARY OF RESULTS 


A summary of scores for all experiments per- 
formed in our laboratory is presented in Table 7.1. The 
frequency distribution for all data is presented in Table 
7.2 and is given graphic representation in Fig. 7.2. The 
customary analyses with respect to daily fluctuations, 
consecutive and card position frequencies are contained 
in Tables 7.3A, 7.4, 7.5 and 7.5A. It will be noted that 
in 12470 runs performed by 332 subjects over a period 
of three years an average score of 5.83 per 25 with the 
highly significant critical ratio of 45.70 has been at- 
tained. 


The frequency distribution for the aggregate data 
displays a single mode at 5 thus failing to reveal the 
diversity in scoring exhibited by the groups (high and 
low-scoring) of which it is composed. The form of the 
distribution does, however, diverge sufficiently from 
the theoretical binomial distribution (y2,.= 3992.12, n = 
13) to establish high probability (P =10~9 that the dis- 
tribution was not generated by the operation of chance 
alone. The reverse distribution, on the other hand, 
yields y2 = 8.92, n = 13, p = .79 thus exhibiting an ex- 
cellent fit to the theoretical chance distribution. 


From Table 7.3A it appears that daily scoring 
fluctuates in somewhat random fashion. It is notewor- 
thy, however, that in the direct series, daily averages 
drop below the chance level only thirty-five times while 
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in the reverse series oscillations above and below the 
chance line occur with almost equal frequency, there be- 
ing 104 above, 108 below. 


Frequencies for consecutive hits again emphasize 
the deviation from chance expectancy exhibited by the 
direct data as contrasted with the close agreement with 
chance expectancy which characterizes the reverse data, 


Since the relationship between card position and 
hit frequency exhibited by Mr. C.J. is so consistent and 
so pronounced as to mask the less marked forms of varia- 
tion displayed by other subjects, the summary data for 
the individual tests have in this analysis been separ- 
ated into several categories representing the perforn- 
ance: 1) of all low-scoring subjects; 2) of all high- 
scoring subjects exclusive of Mr. C.J.; 3) of Mr. C.J. 
exclusive of the EE series; 4) of Mr. C.J. in the EE 
series. Inspection of the various categories fails, 
however, to reveal a significant common tendency. The 
need for the further intensive study of individual high- 
scoring subjects is apparent. 


DISCUSSION 


Among the results of these studies, several seem 
to merit special emphasis. Primary, perhaps, among these 
is the marked contrast exhibited under all analyses be- 
tween the aggregate data for the direct and for the re- 
verse series. It will be recalled that these series are 
derived by subjecting identical experimental data to 
treatments which differ only in that one respects, the 
other violates the subject's intention that his calls 
shall be matched against a given order of the pack. If 
chance alone be credited with producing scoring success, 
indifference to order of operation must necessarily be 
postulated. Actually, however, the direct series yields 
for the 12470 runs ($11,750 trials) a highly significant 
average score (5.83) whereas the reverse series yields 4 
statistically non-significant average (4.98). It is at 
once obvious that the only experimental factor which 
could be associated with this difference is the subject's 
intention that the cards be called in a given order. 


A second point of interest is offered by the con- 
trast maintained under all analyses between the per- 
formance of individual subjects. It is at once evident 
that the discrepancy between direct and reverse series 
derives largely from the scoring of those subjects who 
have been termed ‘high-scoring’. Of 332 subjects tested, 
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16 achieved performances which admit them to such classi- 
fication. That the high scoring of these subjects may 
have resulted from the chance operation of the form of 
selection used by Leuba (6) in his generation of good 
'subjects' is conceivable but difficult of acceptance. 
The presence of 316 subjects with essentially chance per- 
formance suggests that scoring success may not be re- 
garded primarily as a function of the experimenter or of 
the experimental technique. 


Among the performances of the high-scoring sub- 
jects, that of Mr. C.J., our subject extra-ordinary, 
who, throughout 3659 runs covering a period of three 
years, maintained a scoring average of 6.85 with a crit- 
ical ratio of 54.83, would seem to merit special comment. 
Characteristic aspects of the performance of this sub- 
ject include: 


1. marked consistency in daily scoring, with some 
evidence in support of the postulate that scoring success 
may be correlated with physical fitness. 


2. definite improvement in scoring level over 
the initial two-year period with sharp decline followed 
by partial recovery for the third year of the experiment. 


3. success in subjecting scoring effort to arbi- 
trary direction as evidenced in successful shifts from 
the DT to the UT and to the EE technique and in the suc- 
= focusing of scoring effort upon a designated 
pack. 


4. success in achievement of consecutive hits. 
In fact, the scoring advantage of this subject would 
seem to derive almost pepe from this excess of fre- 
quency in consecutive hits of the second and of higher 
orders. 


5. pronounced variation in hit frequency with 
respect to card position. This variation assumes two 
distinct forms. The first, which approximates a linear 
relationship, occurs in the main DT and UT series, and 
in the special variations included under DTy and DTyp- 
The second, represented by a U or W-shaped curve, 





16. The emergence in the aggregate data of an average direct score 
of 5.13 with a C.R. of 3.61 for all low-scoring individual sub- 
jects represents the generation of an apparently significant re- 
sult from the pooling of records which were individually non- 
Significant. Whether this final result represents a statisti- 

artifact, the accumulation of small but persistent record- 
ing error, or the existence of whmewhat wide-spread but mar 
ginal ability to score above chance, is a matter for specula- 
tion. It should be noted that much of the scoring here involved 
derives from the unscreened Preliminary Series. 
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appears in the preliminary (unscreened) DT series, 
throughout the EE series, in the 1938-1939 DT control 
series, and under the deferred knowledge technique in 
both DTp and UTp. The appearance of a new curve of oper- 
ation in the EE series is not disturbing. Rather it of- 
fers added support to the postulate (derived from the 
persistence in the UT series of the direct relationship 
established for the main DT series) that the basic fac- 
tor here involved is physical. To reconcile the find- 
ings for the various DT and UT series is, however, dif- 
ficult. The fact that the U-curve appears when scoring 
is relatively low whereas the linear relationship ob- 
tains when scoring is optimum invites speculation. Pat- 
tern instability associated with inadequacy of data, or 
the existence of several factors operating with varying 
effectiveness might be postulated. The dominance of the 
U-pattern throughout the preliminary and the EE series, 
and of the linear pattern throughout the main DT and UT 
series would suggest limited applicability for the first 
of these hypotheses. The multiple factor hypothesis 
seems, however, to merit further consideration. In sinm- 
plest form, it might involve the postulate that some 
primary and relatively unstable psychological factor 
plays a dominant role when scoring is low, and that a 
secondary and physical factor, associated with card po- 
sition, becomes apparent when this primary factor 
achieves increased stability (thus prgyi din a certain 
constancy or uniformity of background‘). This postu- 
lated dominance and instability of the primary factor 
might be invoked in the endeavor to account for the di- 
verse performance curves of the various high-scoring 
subjects. Essentially, however, the fascinating prob- 
lem of the relation between scoring success and card po- 
sition still stands. 


A final point of interest with reference to this 
topic is presented by the secondary rhythmic fluctua- 


tions which appear throughout the curves of operation of 


Mr. C.J. and of several high-scoring subjects. It is in- 
teresting to recall in this connection the findings of 
Guilford (5) with regard to liminal stimuli, and to spec- 
ulate upon the possible bearing of these findings upon 
scoring pattern. 











17. An electric circuit might offer a crude analogy. With sporad- 
ic (or violently fluctuating) electro-motive force, an inter- 
mittent or fluctuating current would result. The effect of 
the introduction into the circuit of variable resistence would 
be partially masked. With relatively steady electro-motive 
force, on the other hand, the relation between current and re- 
sistence is at once evident. 
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CONCLUSION 


RO 


Throughout this study it has been the purpose of 
the authors to present and to evaluate their results 
against a chance hypothesis (as exemplified theoretically 
by the binomial predictions and empirically by the reverse 4 
series). Under all analyses there has been established t 
for the direct series a unique statistical position with 
respect both to chance predictions and to the reverse 
series. With sensory cues excluded as a scoring factor 
for all screened data, three hypotheses with respect to 
the obtained results are suggested: 1. the results re- 
present the chance occurrence of the highly improbable; 

2. errors of recording imposed upon the operation of 
chance have produced apparently extra-chance results; 3. 
certain individual subjects possess an ability--admitted- 
ly unstable and imperfect--to perceive without known sen- 
sory means. That chance, operating in Boulder, Colorado 
during the years,1937, 1938, and 1939, should be fairly 
represented by these results, or that errors of record- 
ing should exhibit the frequency and the subject-prefer- 
ence necessary to produce these results are hypotheses 
which the individual reader may choose to adopt. To the 
authors, both positions seem untenable. The hypothesis 
that extra-sensory perception occurs seems inescapable. 


ire 
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Table 0.1 


ERROR FREQUENCIES FOR GROUP TESTS (2840 RUNS) 














Type Total Error Positive Error Negative Error Net Effect on 
of Error Abs. Rel. Abs. Rel. Abs. Rel. Scoring Average 
Recording 316 »00445 56 -00079 69 -00097 -.0046 
Chain 37 -00052 11 -00015 1 -00001 -0035 
Inversion 254 -00358 82 00116 16 -00022 -0232 
Miscellaneous 
Scoring 
Marked hits 24 -00034 24 00034 - 0085 
Unmarked hits 177 -00249 177 -00249 -.0623 
Total 808 01138 173 -00244 263 -00369 -.0317 
Table 0.1A 


SCORES OBTAINED BY VARIOUS EXPERIMENTERS 
IN 1000 DT RUNS PERFORMED BY A SINGLE SUBJECT 








Experimenters No. of Runs Total Score Av. /25 
A.B. and J.S. 122 842 6.90 
A.B. and K.M.* 6 39 6.50 
D.M. and assistants A.B. and J.S. 217 1552 7.15 
A.B. 301 2069 6.87 
D.M. 124 754 6.08 
L.L.** 10 69 6.90 
J.S. 210 1508 7.18 
L.S.#* 10 60 6.00 





* Professor Karl F. Muenzinger 
** Assistant Instructor in Psychology 


Table 0.2 


FREQUENCY DISTRIBUTIONS FOR CONTROL SERIES (1000 RUNS) 








Empirical Series 























Coincidences » AAS 4 5 6 7 8 9101112 13 14 15 

Matching Series I 8 28 73 117 198 195 156 112 68 2913 2010 0 

Matching Series II O 30 75 144 184 199 173 916131 7 41 90 0 0 
Theoretical Predictions* 

Matching Hypothesis 4 25 73 137 184 192 160 110 63 3113 5 100 0 

Binomial Hypothesis 4 24.71 136 187 196 163 111 62 2912 4 10 0 O 





*Accurate to nearest. integer 
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PRELIMINARY SERIES: 


Table 1.1 


SUMMARY OF RESULTS 





Direct Series 


Reverse Series 





Subjects No. of Score 


























Av./25 Dev. o C.R.* Score Av./25 Dev. C.R, 
Runs from from 
n.p. n.p. 
Major 
Subjects 
D.W. 301 2258 7.50 753 35.39 21.28 1507 5.01 2 +06 
E.W. 180 1311 7.28 411 27.37 15.02 950 5.27 50 1.82 
C.J. 380 2438 6.43 538 39.76 13.53 1889 4.97 -ll - .28 
B.C. 190 1270 6.68 320 28.11 11.38 995 5.23 45 1.60 
C.B. 340 2071 6.09 371 37.51 9.86 1668 4.90 -32 - 85 
E.A. 180 1008 5.60 108 27.37 3.94 852 4-73 -48 -1.75 
A.B. 120 637 5.30 37 22.35 1.66 576 4.80 -24 -1.07 
J.H. 260 1353 5.20 53 32.90 1.61 1311 5.04 11 33 
D.M. 280 1451 5.18 51 34.13 1.49 1426 5.09 26 «76 
E.J. 110 557 5.06 7~ Qe 233 580 5.27 30 1.40 
Total for 
Major Sub- 
jects 2341 14354 6.13 2649 97.89 26.84 | 11754 5.02 +49 50 
Total for 
Minor Sub- 
jects 830 4269 5.10 119 58.77 2.02 4055 4-90 -95 -1.61 
Grand 
Total 3171 18623 5.80 2768 114.87 24.09 15809 4.98 -46 - .40 
SUMMARY FOR HIGH-SCORING AND FOR LOW-SCORING SUBJECTS 
Subjects No. of Score Av./25 Dev. o CRs Score Av./25 Dev. C.R. 
Runs from from 
n.p. n.p. 
High 
Scorers 1571 10356 6.59 2501 80.85 30.93 7861 5.00 + 6 07 
Low 
Scorers 1600 8267 5.17 267 81.60 3.27 7948 4.97 -52 - .64 








*Critical Ratio = deviation from chance expectancy 





standard deviation for chance 
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Table 1.1A 


PRELIMINARY SERIES: SUMMARY OF RESULTS FOR INDIVIDUAL MINOR SUBJECTS 





Direct Series 





Subjects No. of 





Runs Score Av./25 Dev. C.R. Score Av./25 Dev. C.R. 
from from c. 

a n.p. C.R. n.p. 
C.A. 40 244 6.1 4h 12.90 3.41 223 5.5 23 1.77 
% : C.D. 70 389 5.5 39 17.06 2:23 326 46 <6 =hS2 
32 E.R. 10 62 6.2 12 6.45 1.86 47 pT -3 ~<-.43 
8 L.F. 10 59 5.9 9 645 1.40 46 46 hm 262 
00 B.P. 60 318 5.3 18 15.80 1.14 274 46 “26 <1.64 
35 L.DG. 40 215 5.3 15 12.90 1.16 200 5.0 0 0 
15 R.LaF. 23 126 5.4 11 9.70" 2. 110 4.8 -5 -.51 
1 R.M. 20 110 5.5 10 9.1% - ta 89 Ach “li -1.21 
33 B.N. 10 57 5.7 7 6.45 1.09 42 Ae2 o ¢: =e 
16 M.W. 40 213 5.3 13 12.90 1.01 192 4.8 -8 - .62 
40 V.M. 40 212 5.3 12 12.90 -93 208 5.2 8 -62 
iy J.W. 10 55 5.5 5 6.45 75 48 4.8 -2 - 30 
C.McS. 56 290 5.1 10 15.26 66 255 4-5 -25  - .66 
M.DeB. 30 157 5.2 7 11.17 23. 460 <4. a ae 
50 G.B. 30 157 5.2 7 wait -63 164 5.4 14 1.24 
_ L.L. 10 53 5.3 3 6.45 47 45 4.5 -5 -.77 
R.K. 10 52 5.2 2 6.45 -31 56 5.6 6 92 
L.B. 30 152 5.0 2 Dee 18 132 Ack -18 - .63 
51 A.L: 30 151 5.0 2 12.397 09 159 5.3 + -80 
sai G.D. 30 149 Aav «3% 214.17 = 140 4e7 -10 - .89 
H.P. 25 122 4-8 -3 10.20 - .29 120 5.0 -5 = 49 
40 F.G. 20 95 4.7 =-§ 9.12 - .55 100 5.0 0 0 
ps: K.K. 20 94 it ‘on 9.12 - .66 98 49 -2 = @ 
A.Be. 10 45 45 -5 6.45 <- .77 57 5.7 7 1.10 
WG. 10 43 OS ee 6.45 -1.09 45 4-5 - i = 
- W.K. 40 184 4-6 -16 12.90 <1.24 193 4-8 -7 = 54 
R.W. 36 163 4.5 «<0? “12.04 < 181 5.Q 1 -07 
S.N. 40 181 4-5 -19 12.90 <-1.47 204 5.1 4 -30 
si J.K. 30 121 4-0 -29 11.17 -2.59 159 5.2 9 -80 
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Distribution of Hit Frequencies (Direct) 


Table 1.2 


PRELIMINARY SERIES: 





















































Subject e 3 2 3 4 5 6 7 8 9 10 ll 12 15 14 15 16 17 8 
D.W. 4 . -— ae oon =a oe ae a eS Se SU 
E.W. i 8 ’ 2-2: Bee! ee we. ce ae eS a 
C.J. Se weeeenkewes © @w-seiecéenw se 
B.C. -. ke Se ee Oe oe ee oe 
C.B. a. 4 hr awe es eS Oe mM CS. 

E.A. ‘wa aew & &@ 9 ee 8° Swe 2. eS Se 
A.B. 1 sSMuBRwDeaeiur?,s 5 1 
J.H. -aur_nre%see#saenezn i ss) 
D.M. 262s © 0 3H DW 2 8 . = 
E.J. 5 ‘ta s+. -. -& 4 3 2 - 
Total Major 
Subjects 6 28 87 218 512 570 S42 558 261 152 101 54 34 22 9 4 O 1 
Total Minor 
Subjects 2 20 SO 117 141 170 LO 119 46 34 #18 « § 22=¢«02«21t#~«l lk 
Grand Total 8 48 137 335 455 540 452 458 509 186 114 57 536 22 10 4 O 1 
Binomial 
Predictions 12 75 225 451 592 622 518 351 198 95 %6 15 4 1 0 
High Scorers 4 12 42 119 189 204 217 249 200 124 88 53 35 22 9 4 01 
Low Scorers 4 36 95 216 264 556 255 209 109 62 26 4 8 O 1 
Distribution of Hit Frequencies (Reverse ) 

Subject o il 2 5 4 5 6 7 8 9 10 ll 12 15 14 «15 «(16«O17 

D.W.. 210 1 389 SB S&S 4 35 #16 10 5 § 
E.W. 4 10 24 S31 3 8 19 #10 6 noe. & 2“ & 
C.J. il we © 8 HT CB BG 16 2 1 
B.C. 2 12 2 30 4 S81 2 lu =210 i 2&2 
C.B. 1 56 8 5S 5S SB G2 42 2 2 Sa = 
E.A. 24-enunest#ssi iii 4 2 
A.B. 2s uv wot UE SB Dw 9 4 
J.H. 6 20 30 55’ 55 38 26 18 7 .. + 
D.M. 6 22 42 49 46 42 54 25 8 4 @ 
E.J. 21 10 18 20 16 2 2 4 2 ey 
Total Major 
Subjects 8 52 185 S22 416 440 386 258 159 68 30 12 8 1 9 1 
Total Minor 
Subjects 5 25 60 134 159 169 121 91 SB 21 2.83 8&3 * 
Grand Total 15 75 245 456 555 609 507 5349 217 89 36 15 53 1 O 1 
Binomial 

Predictions 12 75 225 451 592 622 518 351 198 95 36 15 4 1 0 0 
High Scorers 6 586 121 217 279 302 266 165 103 47 18 8 141021 
Low Scorers 7 39 124 239 276 507 241 184 114 42 4187 2 0 0 0 
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Table 1.34 
PRELIMINARY SERIES: DAILY FLUCTUATIONS IN SCORING 
Miss D.W. Miss E. W. 
Date Number Direct Reverse Date Number Direct Reverse 
of Runs Average Average of Runs Average Average 
Feb. 6 10 7.30 5.40 Apr. 12 30 6.56 5.30 
Mar. 6 30 6.83 4.96 Apr. 29 20 745 5.00 
Mar. 11 25 6.28 5.84 Apr. 30 10 4-70 4.20 
Mar. 12 36 7.58 5.30 May 5 20 8.40 5.50 
Mar. 14 60 7.58 4.61 May 10 20 10.40 5.50 
Apr. 3 30 7.66 4.86 May 12 20 7.25 5.10 
June 14 80 8.63 5.05 May 13 10 4-90 5.20 
July 27 30 5.76 4.66 May 17 10 9.60 4.70 
May 28 20 6.05 6.25 
Mr. C.J. June 1 20 7.00 5.10 
Feb. 17 20 5.95 4-45 Miss B.C. 
Feb. 18 10 5.00 5.60 
Feb. 19 20 5.00 5.05 Apr. 30 60 6.85 5.29 
Feb. 20 20 6.00 5.45 May 4 70 6.84 5.20 
Feb. 23 20 5.25 4.65 May 11 10 5.40 5.70 
Mar. 5 30 5.10 4.86 May 17 20 6.75 5.40 
Apr. 5 30 6.80 4.73 May 20 30 6.36 4.93 
Apr. 16 10 5.30 4-90 
Apr. 21 30 5.26 4.80 Mr. C.B. 
Apr. 22 10 6.20 5.10 
Apr. 27 10 5.70 6.20 Apr. 26 10 6.90 4.00 
Apr. 29 20 7.45 5.15 Apr. 28 30 6.00 4.60 
May 4 30 6.36 5.60 Apr. 29 50 6.46 4.86 
May 5 20 8.15 4-40 May 3 40 6.05 4-72 
May 13 10 7.00 3.80 May 5 30 6.53 5.03 
May 19 10 4.50 5.30 May 10 30 7.26 5.00 
May 20 30 8.83 4-70 May 12 60 5.56 5.11 
May 21 30 8.43 5.13 May 17 20 5.70 5.15 
May 27 20 6.05 5.10 May 19 20 4-75 4.08 
May 25 50 6.00 5.00 
Mr. E.A. 
Miss A.B. 
Jan. 10 50 5.60 Abd 
Feb. 9 20 5.65 4.40 Apr. 8 2 5.64 4-74 
Feb. 16 10 7.40 4.00 Apr. 16 10 6.20 5.00 
Feb. 17 20 5.50 4.70 Apr. 19 10 5.40 5.20 
Feb. 18 40 5.63 4-95 May 13 10 4.60 4.20 
Feb. 21 20 4.60 5.05 May 19 20 4.80 5.14 
Feb. 25 10 4.50 5.60 May 21 10 4-90 3.60 
Mar. 17 10 6.90 5.30 May 22 10 5.00 5.00 
May 22 10 5.00 5.10 
June 1 20 5.84 4.84 
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Table 1.34, (cont'd) 





Mr. J.H. 








Date 


Number 


Direct 


Reverse 
of Runs Average Average 


Minor Subjects 








Direct 
Average 











4-75 
4-30 
5.90 
4-30 
5.23 
6.11 
5.55 
5.65 
4.78 
5.10 
5.07 
5.50 
5.13 
5.11 
4.88 
5.50 
5.21 
5.35 
4-90 
4.03 
4-90 
4.93 
5.53 
4.70 
5-44 
5.30 











17 10 4-50 5.30 
23 60 5.36 4-98 
2 40 5.42 5.05 
13 40 5.47 4.85 
14 20 5.05 4-90 
27 20 5.05 5.30 
4 10 5.90 5.30 
19 50 4.76 5.32 
28 10 5.10 4.00 
Miss D.M. 
4 10 5.10 5.30 
2 10 4-40 5.20 
3 30 5.67 5.47 
28 20 4-90 6.15 
29 20 5.25 4-45 
12 10 5.20 4-30 
16 10 5.60 5.30 
19 20 4-35 4-95 
26 30 5.60 5.00 
27 10 4.20 5.70 
1 10 5.10 4.80 
5 10 3.80 5.10 
9 10 4.50 5-40 
2 40 5.40 5.05 
> 20 5.95 5.60 
6 10 5.40. 4.00 
« 24 10 5.40 4-90 
Miss E.J. 

18 50 5.16 5.24 
19 10 4.80 4.60 
20 50 5.02 5.48 
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Table 1.4 
PRELIMINARY SERIES: 
Frequencies for Isolated and for Consecutive Hits 
Subject 1 2 3 4 5 6 7 8 9 
D.W. Direct 1240 308 80 20 15 1?) 1 .¢] .¢) 
Predicted 982 189 36 7 1 0 0 0 0 
Reverse 1040 184 26 4 1 0 Oo 0 ie) 
E.W. Direct 676 150 71 14 9 1 1 1 i?) 
Predicted 588 113 22 4 1 0 0 1?) 1) 
Reverse 590 109 31 8 1 2 re) ie) (¢) 
C.J. Direct 1288 300 106 25 12 3 3 3 1 
Predicted 1240 238 46 9 2 18) 0 0 18) 
Reverse 1263 234 46 5 0 Oo 0 ie) 0 
B.C. Direct 647 179 60 15 5 0 0 0 0 
Predicted 620 119 23 4 1 1?) 1¢) 0 0 
Reverse 600 116 43 7 i¢) 1 re) ie) ie) 
C.B. Direct 1110 307 82 19 5 18) 0 QO i?) 
Predicted 1110 213 4l 8 2 1¢) ie) 0 0 
Reverse 1104 190 49 8 1 0 0 ie) 1?) 
E.A. Direct 623 128 25 5 2 4 0 0 0 
Predicted 588 113 22 a oe ee 
Reverse 570 106 18 4 0 i?) Q 8] i¢) 
A.B. Direct 416 72 23 2 i¢) 0 0 18) 0 
Predicted 392 75 14 3 1 ie) 18) 0 8) 
Reverse 377 70 17 2 i?) i¢) i?) 0 ie) 
J.H. Direct 878 165 36 8 1 fe) fe) te) (e) 
Predicted 849 163 32 6 1 .¢) 0 .¢) 1?) 
Reverse 831 158 39 8 3 0 0 0 0 
D.M. Direct 911 200 36 8 2 1?) 19) 0 18) 
Predicted 914 176 34 6 1 0 0 0 0 
Reverse 902 181 35 13 1 0 0 0 0 
E.J. Direct 356 76 9 3 2 re) 0 0 0 
Predicted 359 69 13 3 ce) i?) fe) 0 e) 
Reverse 359 82 15 3 0 0 ce) 0 ie) 
Total Majors Direct 8135 1885 528 119 53 8 5 4 1 
Predicted 7641 1468 282 54 10 2 0 0 0 
Reverse 7636 1430 319 62 7 3 0 0 .@) 
Total Minors Direct 2769 543 103 21 3 1 0 0 0 
Predicted 2709 521 100 19 4 1 fe) fe) 0 
Reverse 2661 492 94 21 5 2 1 0 0 
Grand Total Direct 10904 2428 631 140 56 9 5 4 1 
Predicted 10350 1989 382 73 ls 3 1 0 0 
Reverse 10297 5 1 0 0 

















; 
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Table 1.5 
PRELIMINARY SERIES: 


Average Scores (per 25) 


For Successive Five Card Groups 




















Subject Direct Reverse 
I II III IV Vv I II III IV V 
D.W. 7.34 6.91 7-52 7.61 6&.12 5.27 4.97 5.02 4.80 4.98 
E.W. 7.89 7.56 7.53 6.97 6.47 4-97 5-44 5.81 5.25 4.92 
C.J. 6.34 6.20 6.46 6.30 6.77 4-78 4.87 5.40 5.12 4.70 
B.C. 6.53 6.50 6.97 6.47 6.95 §.58 5.32 4864 5.37 5.06 
C.B. 6.04 6.13 5.92 6.04 6.31 4-78 4.97 5.04 4-80 4.94 
E.A. 5.17 5.67 5.64 5.39 6.14 4-83 4.61 4.81 4-44 4.98 
A.B. 5.17 4-67 5.54 5275 5242 4-46 4-63 4-75 4.83 5.33 
J.H. 5.35 5.23 4-87 5.62 4.96 AcT7 4-865 5227 §$.-15 5.17 
D.M. 5.00 5-07 5.43 4-98 5.43 $23 4050 5-21 5-29 5.23 
E.J. 4.77 5.32 4-68 5.27 5.27 5-14 5.91 4-86 5.27 5.18 
Total 
major 6.28 6.22 6.37 6.33 6.54 5-15 5.12 5.33 5.20 5.19 
Total 
minor 5-39 5.20 4.87 5.06 5.20 4.80 4.88 4.80 5.14 4.81 
Grand 
total 6.04 5.94 5.96 5.99 6.18 5.06 5.06 5.19 5.19 5.09 
(5-94) (6.05) 
Subject Direct Reverse 
I II III IV V I II III IV Vv 
High 
Scorers 6.54 6.46 6.64 6.49 6.85 5.00 5.00 5.16 4.96 4.91 
(6.35) (6.65) 
Low 
Scorers 5.26 5-15 5-00 5.20 5.22 4-87 4.86 4.95 5.15 5.01 
(5.11) (5.17) 
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Table 1.5A 
PRELIMINARY SERIES: 


Hit Frequency with Respect to Card Position (Direct) 





Card D.W. E.W. C.J. B.C. C.B. E.A. A.B. J.-H. D.M. E.J. Total Total Grand 
Major Minor Total 
(60)* (86) (76) (68) (68) (86) (24) (52) (56) (22) (468) (166) (654) 





1 110 62 102 55 85 45 31 62 58 25 655 197 850 
2 66 61 86 58 85 40 29 55 66 20 566 179 745 
5 90 44 97 45 77 58 17 54 47 18 525 187 712 
+ 100 57 108 48 83 28 19 54 65 20 582 167 749 
5 76 60 89 46 81 55 28 55 44 22 534 165 699 
6 74 52 86 41 61 54 21 52 61 22 514 158 672 
7 96 56 78 49 96 47 22 52 65 25 586 190 776 
8 86 48 95 52 78 44 16 58 56 23 556 178 734 
9 82 59 109 65 70 59 35 54 55 26 588 165 761 
10 78 57 105 42 92 40 20 56 59 21 568 174 742 
l 100 60 81 49 80 39 22 52 55 21 559 152 711 
12 88 60 105 86 44 25 48 67 25 600 151 751 
15 84 589 108 65 85 44 25 51 65 20 600 165 765 
14 87 50 94 45 85 34 38 57 67 19 574 178 752 
15 94 42 105 54 71 42 25 45 20 546 164 710 
16 85 49 95 44 87 55 50 59 52 26 558 180 738 
17 100 55 94 60 95 37 28 75 60 50 652 166 798 
18 98 52 79 48 75 54 21 61 52 25 545 180 725 
19 92 45 104 47 85 45 50 57 55 19 577 157 734 
20 85 50 86109 47 71 43 29 42 62 16 554 157 711 
21 98 52 88 57 100 44 29 52 69 19 608 186 794 
22 78 45 101 55 91 45 26 50 60 23 574 177 761 
23 92 44 104 50 95 51 22 65 56 26 585 175 758 
24 87 41 106 o4 72 40 16 55 58 25 550 140 690 
25 154 Sl 116 48 71 61 37 40 61 25 644 185 829 





* Chance expectancy for each card position (.2N). 
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Table 1.54 
PRELIMINARY SERIES: 
Hit Frequency with Respect to Card Position (Reverse ) 

Card D.W. E.W. C.J. B.C. C.B. EA. A.B. J.H. DM. E.J. Total Total Grand 

Major Minor Total 

(60)* (66) (76) (38) (68) (56) (24) (52) (56) (22) (468) (166) (654) 
1 71 45 65 28 60 35 18 54 55 20 451 157 608 
2 65 55 72 56 78 29 24 44 56 50 474 159 635 
t) 54 34 80 55 63 44 21 42 57 22 470 162 652 
a 62 85 69 41 68 26 24 65 65 19 470 157 627 
5 67 32 70 4 56 38 20 45 62 22 466 161 627 
6 45 58 69 39 64 45 24 55 60 28 465 176 6539 
7 69 41 72 47 66 21 21 55 438 27 465 164 629 
8 61 42 85 47 82 46 22 58 45 27 515 155 668 
9 62 56 79 55 56 24 20 42 §1 27 452 158 590 
10 62 39 65 34 70 32 24 46 50 21 445 157 600 
ll 41 37 72 26 80 58 20 61 55 18 451 174 622 
12 66 40 68 51 57 81 25 47 6a 25 452 157 609 
15 84 58 108 65 85 44 25 $1 65 20 600 165 765 
4 55 58 80 24 55 $1 16 63 54 25 459 145 584 

15 58 35 82 40 68 29 50 52 56 19 469 158 

16 62 87 74 44 78 36 ah 58 65 2 500 176 676 
17 47 56 85 42 58 27 16 49 57 26 44. 172 615 
18 71 46 77 44 of 54 50 55 65 19 491 176 667 
19 47 32 83 28 71 32 27 57 $1 21 449 164 615 
20 62 58 72 46 65 51 19 51 62 26 472 165 657 
21 59 33 65 37 66 55 25 61 57 26 460 160 620 
22 4 51 77 39 60 38 24 43 60 22 455 152 605 
25 57 41 80 42 77 40 $1 59 61 21 509 176 676 
24 67 52 72 44 55 55 28 55 62 26 462 171 655 
25 65 40 65 51 80 51 22 48 62 19 462 140 602 





* Chance expectancy for each card position (.28). 
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Table 1.5A 
PRELIMINARY SERIES: 


Card Position Summary for High and for Low-Scoring Subjects: 








Card High Scorers Low Scorers High Seorers Low Scorers 





(314) (320) (314) (320) 
1 457 373 304 304 
2 396 349 320 313 
3 389 323 328 304 
4 424 325 301 326 
5 387 312 317 310 
6 368 304 298 341 
7 422 354 316 313 
8 403 331 363 305 
g 422 329 292 298 
10 412 330 302 298 
11 409 302 294 328 
12 437 314 293 316 
13 441 322 441. 322 
4 393 359 281 303 
15 406 304 312 315 
16 391 347 331 345 
17 44h 357 293 320 
18 384 339 326 341 
19 418 316 293 320 
20 405 306 314 323 
21 439 355 293 327 
22 415 336 299 306 
23 416 342 337 348 
24 400 290 303 330 
25 481 348 310 292 





*Chance expectancy for each card position (.2N) 
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Table 2.1 


GROUP SERIES: SUMMARY OF RESULTS 














Direct Series Reverse Series 
No. of Score Av./25 Dev. from C.R. Score Av./25 Dev. from C.R, 
Runs n.p. n.p. 
2840 14187 4.995 -13 108.71 -.12 14049 4.95 -151 -1.99 
Table 2.2 


GROUP SERIES: DISTRIBUTION OF HIT FREQUENCIES 





Mts per 0 1 #2 3 4 5 6 7 8 , W@W i 2. 3258 
Run 





marect 35 7 190 404 9528 572 415 335 168 90 32143 2 8 
Reverse 17 69 200 414 532 562 421 314 186 80 30 12 3 





Table 2.3A 


GROUP SERIES: DAILY FLUCTUATIONS IN SCORING 





Date Number Direct Reverse 
of Runs Average Average 








Apr. 19 470 5.04 4.87 

Apr. 20 590 4-91 5.01 

Apr. 22 780 5.03 4.81 

Oct. 25 450 4.79 5.08 

Oct. 27 550 4-97 4.98 
Table 2.4 


GROUP SERIES: FREQUENCIES FOR 
ISOLATED AND FOR CONSECUTIVE HITS 





1 2 3 - 3s 





Direct 9341 1705 331 90 13 
Predicted 9270 1781 342 65 13 
Reverse 9295 1750 335 65 13 4 


now 





Table 2.5 


GROUP SERIES: AVERAGE SCORES (PER 25) 
FOR SUCCESSIVE FIVE CARD GROUPS 





Series I II III IV v 





Direct 4.97 4.94 5.06 4.94 5.07 
Reverse 4.87 4.94 4.92 5.00 5.00 








i ree ee 


Dee Ye 


PEE SPA 
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Table 2.5A Table 2.6 


GROUP SERIES: CARD POSITION FREQUENCIES GROUP SERIES: DISTRIBUTION OF TOTAL 

















a INDIVIDUAL SCORES 
Card Direct Reverse 
aa Individual Frequencies 
1 583 538 Scores Direct Reverse 
2 584 557 
-_ 3 525 586 30 0 1 
9 4 602 535 31 0 0 
Be a 527 548 32 0 0 
33 0 0 
6 551 559 34 1 1 
7 608 576 35 0 3 
8 54d 564 36 3 5 
- 9 558 521 37 1 3 
10 547 586 38 3 0 
39 3 6 
~ ll 595 553 40 4 10 
12 551 546 41 11 4 
13 578 578 42 8 6 
aa 4 566 557 43 6 8 
15 586 561 44 22 12 
45 14 8 
16 584 535 46 15 15 
17 567 558 47 16 17 
18 548 602 48 18 15 
19 568 571 49 14 18 
20 537 576 50 19 19 
51 21 25 
21 543 554 52 13 20 
22 593 624 53 9 19 
23 548 544 54 10 15 
24 581 567 55 21 11 
25 613 553 56 8 8 
57 12 8 
58 3 7 
59 5 8 
60 6 3 
61 2 6 
62 3 1 
63 2 0 
64 5 0 
65 0 0 
66 * 2 0 
67 3 0 
68 (¢) 0 
69 1 1 
70 0 1 
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MAIN DT SERIES: 


Table 5.1 


SUMMARY OF RESULTS 





























Direct Series Reverse Series 
Subjects No. of Score Av./25 Dev. o C.R. Score Av./25 Dev. C.R. 
Runs from from 
n.p. n.p. 
Mejor Subjects 
C.J. 1000 6895 6.89 1895 64.50 29.55 5055 5.05 55 61 
R.S. 500 5015 6.05 515 45.61 11.25 2511 5.02 lu 24 
C.B. 700 5852 5.50 552 54.02 6.52 5554 5.05 34 -65 
D.M. 500 2584 5.17 84 45.61 1.84 2414 4.85 -86 -1.89 
H.D. 500 2550 5.06 50 45.61 -66 2440 4.88 -60 -1.52 
Total Major 
Subjects 5200 18872 5.90 2872 115.40 24.89 15932 4.98 -68 - .59 
Minor Subjects 
8.B. 100 624 6.24 124 20.40 6.07 481 4.81 -19 - .95 
G.E. 100 622 6.22 122 20.40 5.98 490 4.90 -10 - .49 
J.M. 120 710 5.92 110 (22.35 4.92 624 5.20 24 1.07 
L.L. 140 764 5.46 64 24.15 2.65 676 4.85 -24 - .99 
C.L. 150 672 5.17 22 235.26 94 625 4.81 -25 -1.07 
J.R. 140 715 5.09 15 24.15 54 751 5.22 31 1.28 
J.H. 100 500 5.00 0 20.40 -00 486 4.86 -l4 - .69 
E.C. 150 745 4.97 - 5 24.98 - .20 756 4.91 -l4 - .56 
Total Minor 
Subjects 980 5550 5.46 450 65.85 7.04 4849 4.95 -51 - .80 
Series Total 4180 24222 5.79 58522 151.89 25.19 20781 4.97 -119 - .90 
SUMMARY FOR HIGH-SCORING AND FOR LOW-SCORING SUBJECTS 
Direct Series Reverse Series 
Subjects No. of Score Av./25 Dev. C.R. Score Av../25 Dev. C.R. 
Runs from from 
B.p. nD.p. 
High Scorers 2660 16478 6.19 5178 105.21 50.21 15549 5.02 49 47 
Low Scorers 1520 7744 5.09 144 79.55 1.81 7432 464.90 -168 -2.l1 
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Table 5.2 
MAIN DT SERIES: DISTRIBUTION OF HIT FREQUENCIES 
Subjects e © 2 5 4 § 6 7 8 8 10 lL 12 15 14s 16 16 
C.J. Direct 2415 25 S53 87 #107 152 158 157 108 62 40 20 10 56 21 
Reverse 6& 24 70 116 215 176 162 106 71 8 l 7,2 000 0 
R.S. Direct os B&B 82 8B 2B. 2 &2. 2 a2 Bw oe i 
Reverse 110 535 64 101 2 85 S&S 385 415 S: & de © 
C.B. Direct 0 16 «4 56 110 18% 117 110 61 8&lSw 6 .l1ftll 
Reverse 115 55 96 155 115 120 86 48 2 MM @& fh ee 
DM. Direct . tt fa Bw TF a. eS Sw ae 2 
Reverse 2 15 539 80 9 95 72 5 28 lz , 
H.D Direct 218 S2 56 85 106 82 65 28 2 ~ ws 
Reverse 4 16 27 78 101 102 72 5S 29 li 4 8 6& 
Major Direct 6 6 145 276 427 518 528 495 5536 209 968 65 21 12 6 2@ il 
Total Reverse 16 78 226 454 646 S582 Sll 349 211 92 &1177000i(+#0 
8.B. Direct o.6hCU€~*# x 2 82... = 6 § 2 
Reverse 1 1 ee &2 2 2 ae 5 2 2 0 
G.E. Direct i?) 1 sk. Bo eR eS 8 § 2 
Reverse 2 5 a i oe ae ae 6 $ 0.60 
JM. Direct i ’.i.- BB. 2 &¢ wre 7 § 1 
Reverse O 4 ao Mw a Ww Me 9 = 
L.L. Direct e 2 4 20 25 25 22 25 158 8 = 
Reverse 1 2 19 16 22 3 ll iu 7 ojou 
C.L. Direct » -& Tk SR we he 8 6 ec = @ 2 
Reverse 1 5 i: & iB 7 1 0 0 Oo 0 
JAR. Direct ; ¢€. 23.86 &.2 ww eS 5 a = 
Reverse 1 . FF £1 Se ae.lUe 6 a. %§ 
E.C. Direct a a a fy ak ee oe 4 5’ 60 
Reverse 0 2 15 19 Sl S6 19 18 § a - 2 
J.H. Direct e 2 . f=. @ ee aa 9 6 a 2 0 0 
Reverse O 1 tl 14 25 8 27 5 2 4 .. oe 
Minor Direct 5 14 45 118 159 162 167 125 8 47 27 #7##41¢60#i4i1 
Total Reverse 5 17 79 126 189 211 151 102 S& 585 8 a. 2 & 
Series Direct 9 75 168 594 585 701 695 618 420 256 125 70 22 12 7 21 
Totel Reverse 21 95 505 560 855 795 662 451 265 125 589 21 8 O OF O 0 
MAIN DT SERIES: DISTRIBUTION SUMMARY FOR HIGH-SCORING AND FOR LOW-SCORING SUBJECTS 
High Direct 2 38 95 191 522 399 450 427 551 194 105 65 22 12 6 2 1 
Scorers Reverse 14 57 202 550 555 491 426 295 185 87 25 15 4 
Low Direct 7 37 95 2058 265 SO2 245 191 88 62 20 5 0021 
Scorers Reverse 7 58 105 250 S02 S502 236 156 82 38 16 6 4 0 O 
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Table 3.3A 
MAIN DT SERIES: DAILY FLUCTUATION IN SCORING (MAJOR SUBJECTS) 
Date Number Direct Reverse Date Number Direct Reverse Da 
of Runs Average Average of Runs Average Average 
Mr. C.J. Oct. 25 40 4.68 5.08 
Sept. 30 30 5.60 5.00 Get. 26 60 5-79 4-90 det. 
Nov. 2 40 5.40 4.98 No 
Oct. § 50 6.98 5.56 Ve 
Nov. 9 60 5.58 4.71 N 
Oct. 6 50 5.76 5.24 lov. 
Nov. 16 40 6.05 5.12 —— 
Get. 7 50 6.76 5.20 Nov. 22 30 5.77 3.10 
Oct. 8 40 6.17 4.95 . 
Oct. 12 80 7.91 5.31 — 
Oct. 13 40 6.95 5.20 Mr. H.D. Apr. 
Oct. 14 80 7.47 4-97 Apr. 17 30 5.60 5.33 ond 
Oct. 15 30 6.90 4.66 y 
Apr. 20 20 5.35 5.05 Ma: 
Oct. 18 20 6.55 5.10 y 
Apr. 21 30 4.23 5.13 Oct 
Oct. 19 50 6.36 5.26 ° 
Apr. 24 20 5.00 4.75 —— 
Oct. 20 10 8.90 4.70 
Apr. 25 130 5.20 4.71 
Oct. 21 80 7.78 4-98 
Apr. 26 10 5.90 6.00 
Oct. 25 20 5.60 4.10 
Apr. 27 30 5.13 4.9 Nov 
Oct. 26 10 5.00 4.60 . 
May 2 30 4.60 5.00 Feb. 
Oct. 28 50 5.74 5.08 
May 4 70 5.01 5.11 Mar. 
Oct. 29 30 7.76 5.73 
May 9 40 4-92 4.78 Mar. 
Nov. 2 40 5.77 4.85 
May 18 30 5.23 4.30 Apr. 
Nov. 3 20 5.65 4-90 pea 60 4.93 4.68 pr 
Nov. 4 20 6.10 4-65 . 
Nov. 9 30 6.70 4.60 
Nov. 10 70 7.54 4.61 mee 5.A. 
Nov. 11 100 7.49 5.03 Oct. 8 15 6.00 4.40 fee. 
Oct. 11 15 5.93 3.87 May 
a ee Oct. 13 20 5.35 5.20 May 
Oct. 7 10 5.50 4.90 Oct. 26 10 5.80 4.50 May 
Oct. 28 30 4-63 5.40 May 
Oct. 13 30 5.10 4.93 
Nov. 1 20 5.20 4-50 — 
Oct. 15 40 5.43 5.14 
Nov. 4 30 4.87 4.13 
Oct. 18 60 6.68 5.22 
Nov. 6 20 4.80 5.25 
Oct. 20 60 6.05 5.38 
Nov. 11 30 5.13 4.60 
Oct. 25 50 6.80 4.88 
Nov. 15 20 5.30 4.85 
Oct. 27 40 6.18 4-83 
Nov. 16 6 4.00 4.33 
Nov. 1l 80 6.45 4.82 
Nov. 20 20 4.85 5.20 
Nov. 3 30 5.23 4.73 
Dec. 6 10 4.80 5.40 
Nov. 5 40 5.58 §.15 
Feb. 3 10 5.80 4.60 
Nov. 12 20 5.60 4-70 
Apr. 5 4 5.00 6.50 
Nov. 19 10 4.30 5.40 
Nov. 24 30 6.20 5.10 Apr. 19 10 6.50 5.60 
; Z ~ Apr. 20 30 5.20 4.60 
Apr. 22 20 5.65 5.10 
Mr. C.B. 
r. C.B Apr. 26 20 5.05 4.95 
Oct. 1 10 4-40 4.80 ~~ od 4.80 4-00 
a June 1 20 5.20 5.20 
Oct. 5§ 70 5.23 §.13 
June 6 20 5.05 4.40 
Oct. 7 60 4-93 5.03 
June 7 10 4.70 4.60 
Oct. 1l 70 4.83 4.83 
June 16 20 5.80 5.30 
Oct. 12 60 4-93 5.47 
June 19 40 4-70 4.85 
Oct. 19 80 6.49 5.01 A 10 0 5.23 90 
Oct. 21 60 6.28 5.30 sal 4 ° Ae 
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MAIN DT SERIES: 


Table 3.3A (cont'd) 


DAILY FLUCTUATIONS IN SCORING (MINOR SUBJECTS) 






























































Date Number Direct Reverse Date Number Direct Reverse 
of Runs Average Average of Runs Average Average 
Mr. S.B. Miss E.C. 
Oct. 29 20 6.59 4-50 Sept. 28 20 4-45 4.50 
Nov. 5 40 6.00 4.98 Sept. 30 20 5.50 4.90 
Nov. 12 40 6.13 4.80 Oct. 3 10 4.70 5.60 
Oct. 5 40 5.10 4.68 
Mr. G.E. Oct. 7 10 4.70 5.50 
Oct. 8 10 4.80 6.00 
Apr. 29 30 6.16 5.00 Oct. 12 15 5.20 4-20 
May 9 20 6.15 4.65 Oct. 19 25 4.88 5.08 
May 23 20 6.30 4.70 
May 25 20 6.55 5.00 Mr. J.H. 
Oct. 27 10 5.70 5.60 
Dec. 2 10 5.00 4-30 
Miss L.L. Dec. 3 30 4.73 5.22 
Dec. 5 30 4-90 4.70 
Nov. 10 30 5.87 4-43 Dec. 6 20 5.40 4-35 
Feb. 23 10 5.80 5.20 Dec. 13 10 5.30 4.80 
Mar. 1 10 5.80 3.70 
Mar. 8 60 5.48 §.27 
Apr. 27 10 5.10 5.00 Miss C.L. 
Apr. 28 20 4.60 4.40 
Oct. 25 20 6.30 4-50 
Miss J.M. Oct. 27 40 5.00 5.23 
Oct. 28 30 5.23 4.63 
Apr. 28 40 5.90 5.17 Nov. 4 40 4.73 4.68 
May 12 30 5.56 5.16 
May 23 10 7.80 4-50 Mr. J.R. 
May 24 20 6.20 5.65 
May 25 20 5.25 5.20 Jan. 24 30 4-40 5.26 
Jan. 26 20 5.30 5.40 
Feb. 1 30 5.20 4.53 
Feb. 6 30 5.53 5.56 
Feb. 10 30 5.10 5.40 
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Table 3.4 

MAIN DT SERIES: FREQUENCIES FOR ISOLATED AND FOR CONSECUTIVE HITS 
Major Subjects 1 =. . oS 69. 8 Sw ee 
Direct 3371 O81 259 36 DB WwW £6 0 VC 8 Ba 
Jencks Deefteteit 3264 627 120 23-4 10 «0 08 © Sia 
Reverse 3379 oe i123 26° 6 #2 i 0 606 8 Gam 
Direct 1687 423 96 29 9 4 1 0 0 0 0 0 
Simmons Predicted 1632 314 60 12 2 0 0 0 0 0 0 0 
" Reverse Se a a: a i i ce Ae 
Direct 2323 512 102 30 11 4 0 0 0 1¢) 0 0 
Bilz Predicted 2285 439 84 16 3 es. 8 °o 0 =a 
Reverse ae ae GS tiC eC COCO ee 
Direct 1718 291 69 9 3 2 2 0 0 0 0 0 
Martin Predicted 1632 314 60 12 2 0 1@) 0 0 0 0 (0 
Reverse 1628 301 48 10 eS 8@© 6 © © @€. aaa 
Direct 1631 316 8666 2 2 1 0 0 ¢ Baa 
Dixon Pregietet 16352 324 CO @W 20 06 0 0 8 Gam 
Reverse 1622 287 60 9 3 1 1 0 0 0 0 0 
Total Direct 10930 2493 592 153 64 30 8 0 0 0 o 4 
Major Predicted 10445 2007 383 7h 1s 3 1 0 0 0 Oo 0 
Subjects Reverse 10624 1894 383 7a. me 3 2 ce) & © Sa 
Minor Subjects 1 2 3 4 8 .6 7%. 8 93 »*» 2a 
Direct 368 91 17 3 1 a 0 0 0 0 0 0 
S.B. Predicted 326 63 12 2 0 0 0 0 0 0 0 0 
Reverse 310 68 8 > 2 °o 08 © © 868 - iam 
Direct 356 82 24 5 2 0 0 0 0 i¢) 0 0 
G.E. Predicted 326 63 12 2 0 0 0 0 0 0 0 «(0 
Reverse 333 62 9 1 0 0 i¢) 0 0 0 0 0 
Direct 430 88 24 4 2 1 0 0 1¢) 0 o>.- 
J.M. Predicted 392 75 14 3 1 1¢) 1¢) 0 1?) 0 Oo 0 
Reverse 458 48 22 1 0 0 Oo O 0 + £m 
Direct 472 104 20 6 0 0 0 0 1¢) 0 Oo 0 
L.L. Predicted 457 88 17 3 1 0 .¢) 0 0 0 0 = 
Reverse 411 85 23 4 2 0 0 0 0 0 0 4 
Direct 432 65 22 7 2 1 0 0 0 0 = 
C.L. Predicted 424 81 16 3 1 1¢) 1?) 1¢) 0 .¢) 0 0 
Reverse 414 77 12 4 1 0 1?) 10] 1¢) 1?) 0 0 
Direct 456 84 25 2 0 1 0 0 0 0 0 0 
J.R. Predicted 457 88 17 3 1 0 0 0 0 0 Oo 0 
Reverse 495 80 24 i © @ 6 © 68 © ae 
Direct 338 52 12 3 2 1?) ¢) 0 1¢) 0 0 0 
J.H. Predicted 326 63 12 2 6@ & @ @© 6 6fFO=a- 
Reverse 339 46 10 | a 0 0 0 0 0 0 (0 
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Table 3.4 (cont'd) 


























Minor Subjects 1 2 3 4 5 6 7 8 >. ae S .ae 
Direct 462 103 23 2 ie) 18) 0 i?) ie) 0 0 .@] 

E.C. Predicted 490 94 18 3 1¢) 12) 1¢) 1?) .¢) 1) 0 1?) 
Reverse 550 59 16 § 0 0 0 0 0 0 0 0 

Total Direct 3314 669 167 32 9 4 0 .¢) 0 0 0 0 
Minor Predicted 3199 615 118 23 4 1 18) 0 0 0 1] 16] 
Subjects Reverse 3311 526 124 21 6 0 0 0 0 0 0 0 
Total Direct 9702 2251 542 152 64 28 5 0 i@) .¢] 0 @) 
High Predicted 8682 1668 320 61122 00000 090 
Scorers Reverse 8887 1570 337 58 12 2 1 0 ie) ie) 0 0 
Total Direct 5037 911 217 23 9 6 3 0 0 0 0 0 
Low Predicted 4961 953 183 35 TCR. a Gt a eee 
Scorers Reverse 5048 850 170 34 5 1 1 0 i¢) 0 0 0 
Series Total Direct 14244 3162 799 3145 73 MB S& OO 8 8 hlU8HlhlU 
Predicted 13644 2622 503 96 #18 4 1 1¢] 0 0 0 0 

Reverse 13935 2420 507 92 17 3 2 i¢) 0 0 8) 0 
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Table 3.5 
MAIN DT SERIES: AVERAGE SCORES (PER 25) FOR SUCCESSIVE FIVE CARD GROUPS 
Subjects Direct Reverse 
I II III IV Vv I II III IV v 
Major Subjects 
R.S. 5.88 6.12 6.24 6.09 5.80 5.09 4-70 5.18 5.24 4.90 
C.B. 5.52 5.57 5.56 5.73 5.14 5.08 5.06 5.10 5.07 4.93 
H.D. 5.09 4.80 5.27 4.95 5.19 5.02 4.86 4.93 4.95 4.64 
D.M. 5.02 5.30 4-91 5.23 5.38 4-85 4.69 4.91 4-78 4.91 
Total 5.39 5-46 5.50 5.52 5.35 5.02 4-85 5.04 5.02 4.81 
C.J. 7.66 7.31 7.07 6-37 6.07 4-82 5.24 5.07 5.21 4.84 
Major Total 6.10 6.04 5.99 5.79 5.58 4-95 4.97 5.05 5.08 4.85 
Minor Subjects 
8.B. 6.15 6.55 6615 6.10 6.25 4e95) 4665 5615 4.55 4e75 
G.E. 6.35 5.65 6.30 7.05 5.73 4-85 5.35 5.05 4-05 5.20 
J.M. 5.75 5.93 5-79 6.38 5.75 5.46 5.46 5.63 5.00 4.46 
L.L. 5.61 6.00 5.43 5.25 5.00 5.50 4.04 4-89 5.11 4.61 
C.L. 4-88 5.38 4-77 5.31 5.50 5.12 4-58 4.58 4.89 4.89 
J.R. 4e93 5429 5643 5.18 4.64 5.50 5.32 5.14 4.68 5.46 
J.H. 5.00 4.65 4675 4-95 5.65 5-20 4-55 5.00 5.20 5.35 
E.C. 5.33 4-87 heT7 5-27 4-60 5-43 5.93 4,60 4.83 4.73 
Minor Total 5.46 5.52 5.38 5.63 5.32 5.28 4.85 4.98 4.70 4.92 
High Scorers 
Major Subjects 5.67 5.80 5.84 5.88 5.41 5.08 4.91 5.13 5.14 4-92 
(without C.J.) 
Minor Subjects 5.92 6.02 5.87 6.12 5.63 5623 4-83 $17 473 4:35 
Total High Scorers 
(without C.J.) 5.74 5-86 5.85 5.95 5.47 5.12 4.89 5.14 5.03 4.86 
Total High Scorers 
(with C.J.) 6.46 6.41 6.31 6.10 5.70 5.01 5.02 5.30 5.10 4.65 
Low Scorers 
Major Subjects 5.06 5.05 5.09 5.09 5.29 4-92 4.78 4.92 4.87 4.78 
Minor Subjects 5.05 5-07 4-95 5.19 5.04 5.33 4.88 4.82 4.68 5.09 
Total Low Scorers 5.05 5.06 5.04 5.03 5.20 5.07 4.81 4.88 4.80 4.88 
Grand Total 
(without C.J.) 5.41 5.47 5-46 5.55 5.33 5.10 4.85 5.02 4.90 4.87 
Series Total 5-95 5.91 585° 5.75 5.54 5-03 4-94 5.03 4-98 4.86 
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Table 3.5A 
MAIN DT SERIES: 
Hit Frequency with Respect to Card Position (Direct) 
Major Subjects 
Card R.S C.B. H.D. D.M. C.J. High Low Total 
Scorers Scorers 

1 118 164 109 103 301 583 212 795 

2 114 152 103 98 317 583 201 784 

3 103 150 93 78 324 577 171 748 

4 106 141 102 108 298 545 210 755 

5 147 166 102 115 291 604 217 821 

6 130 158 96 95 304 592 191 783 

7 110 158 99 104 274 542 203 745 

8 123 153 96 107 297 573 203 776 

4 126 158 94 111 278 562 205 767 
10 123 153 95 113 309 585 208 793 
11 103 153 99 94 302 558 193 751 
12 139 149 111 87 296 584 198 782 
13 114 153 113 102 277 544 215 759 
4 138 168 101 109 258 564 210 77h 
15 130 155 103 99 281 566 202 768 
16 119 155 90 97 264 538 187 725 
17 125 152 103 99 255 532 202 734 
18 120 170 107 107 238 528 214 742 
19 111 148 93 102 250 509 195 704 
20 134 177 102 118 266 577 220 7197 
21 128 163 96 108 239 530 204 734 
22 122 150 97 112 241 513 209 722 
23 122 134 122 108 228 484 230 714 
24 97 139 110 99 275 511 209 720 
25 111 133 94 111 230 474 205 679 














JOURNAL OF PARAPSYCHOLOGY 








Table 3.54 


MAIN DT SERIES: 


Hit Frequency with Respect to Card Position (Reverse) 


Major Subjects 








Card R.S. C.B. H.D. D.M. C.J. High Low Total 
Scorers Scorers 

1 111 161 90 81 186 458 171 629 
2 104 139 111 98 201 444 209 653 
3 105 138 84 102 200 443 186 629 
4 91 139 105 108 181 411 213 624 
5 98 134 112 96 195 427 208 635 
6 99 152 91 84 214 465 175 640 
7 80 142 98 108 198 420 206 626 
8 94 144 95 93 213 451 188 639 
9 96 138 91 106 205 439 197 636 
10 101 133 111 78 217 451 189 640 
1l 105 161 87 106 187 453 193 646 
12 91 128 90 95 188 407 185 592 
13 114 153 113 102 277 544 215 759 
14 110 134 94 96 184 428 190 618 
15 98 138 109 92 178 414 101 615 
16 100 153 86 92 219 472 178 650 
17 106 124 112 87 201 431 199 630 
18 99 146 104 102 204 449 206 655 
19 115 154 92 101 201 470 193 663 
20 104 133 101 96 217 454 197 651 
21 90 154 91 99 187 431 190 621 
22 101 121 103 115 197 419 218 637 
23 106 139 95 82 184 429 177 606 
24 101 135 82 114 196 432 196 628 
25 92 141 93 81 203 436 174 610 
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Table 3.5A 
MAIN DT SERIES: 
Hit Frequency with Respect to Card Position (Direct) 


Minor Subjects 








Card S.B. G.E. L.L. J.M. E.C. J.H. C.L. JR. High Low Total 
Scorers Scorers 
1 28 23 22 28 34 18 33 26 101 111 212 
2 31 29 38 31 26 23 26 32 129 107 236 
3 22 19 32 24 35 25 12 30 97 102 199 
4 23 32 40 20 30 12 29 23 115 94 209 
5 19 24 25 35 35 22 27 27 103 111 214 
6 20 25 33 33 26 17 30 38 111 111 222 
7 29 24 28 34 31 20 29 28 115 108 223 
8 26 18 39 18 27 21 28 23 101 99 200 
g 34 24 36 28 25 15 30 22 122 92 214 
10 22 22 32 29 37 20 23 37 105 117 222 
11 27 23 37 29 30 18 30 27 116 105 221 
12 23 33 36 28 36 15 21 27 120 99 219 
13 24 22 24 29 27 23 28 26 99 104 203 
4 21 23 29 23 26 16 19 44 96 105 201 
15 28 25 26 30 24 23 26 28 109 101 210 
16 18 31 33 27 28 20 23 23 109 94 203 
17 22 27 22 32 24 12 29 25 103 90 193 
18 30 34 30 34 33 14 31 33 128 111 239 
19 27 25 23 29 40 25 31 28 104 124 228 
20 25 24 39 31 33 28 24 36 119 121 240 
21 28 15 31 23 24 25 29 28 97 106 203 
22 26 27 34 32 31 24 36 35 119 126 245 
23 25 30 24 26 31 26 23 19 105 99 204 
24 22 21 32 22 33 18 26 28 97 105 202 


25 24 22 19 35 19 20 29 20 100 88 188 
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Table 3.5A 


MAIN DT SERIES: 


Hit Frequency with Respect to Card Position (Reverse) 


Minor Subjects 








Card S.B G.E. JM. EC. J.-H. C.L. J.R. High Low Total 
Scorers Scorers 

1 24 15 31 27 40 19 22 32 97 113 210 
2 22 25 31 28 25 24 28 36 106 113 219 
3 22 19 41 23 34 20 28 34 105 116 221 
4 15 20 23 26 29 18 30 27 84 104 188 
5 16 18 28 27 35 23 25 25 89 108 197 
6 24 20 22 27 30 21 24 28 93 103 196 
7 15 20 22 2 30 16 26 34 81 106 187 
8 22 24 16 32 25 22 19 26 94 92 186 
9 17 21 33 26 29 16 23 31 97 99 196 
10 15 22 20 22 34 16 27 30 719 107 186 
11 19 22 35 22 26 11 25 3 98 95 193 
12 20 14 20 34 24 25 22 27 88 98 186 
13 24 22 24 29 27 23 28 26 99 104 203 
14 23 23 29 31 30 23 22 27 106 102 208 
a5 17 20 29 19 31 18 22 31 85 102 187 
16 17 17 29 26 28 20 31 30 89 109 198 
17 19 10 31 25 28 20 19 27 85 94 179 
18 18 15 26 25 34 19 27 26 84 106 190 
19 21 20 25 24 29 15 23 19 90 86 176 
20 16 19 32 20 26 10 27 29 87 92 179 
21 17 28 21 18 27 22 23 31 84 103 187 
22 23 17 33 24 26 21 31 29 97 107 204 
23 21 11 27 23 30 25 27 29 82 111 193 
24 13 20 29 18 27 19 29 22 80 97 177 
25 21 28 19 24 32 20 17 42 92 111 203 
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Table 3.5A 
MAIN DT SERIES: 


Card Position Summary for High- and for Low-Scoring Subjects 








Card High-Scoring Subjects Low-Scoring High-Scoring Subjects Low-Scoring 








with C.J. without C.J. Subjects with C.J. without C.J. Subjects 
1 684 383 323 555 369 284 
2 712 395 308 550 349 322 
3 674 350 273 548 348 302 
4 660 362 304 495 314 317 
5 707 416 328 516 321 316 
6 703 399 302 558 344 278 
7 657 383 311 501 303 312 
8 674 377 302 545 332 280 
9 684 406 297 536 331 296 
10 690 381 325 530 313 296 
11 674 372 298 651 264 288 
12 704 408 297 495 307 283 
13 643 366 319 643 366 319 
uu 660 402 315 534 350 292 
15 675 394 303 499 321 303 
16 647 383 281 561 342 287 
17 635 380 292 516 315 293 
18 656 418 325 533 329 312 
19 613 363 319 560 359 279 
20 696 430 341 541 324 289 
21 627 388 310 515 328 293 
22 632 391 335 516 319 325 
23 589 361 329 511 327 288 
24 608 333 314 512 316 293 


25 574 344 293 528 325 285 
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Table 4.1 


UT SERIES: 


Summary of Results for Direct, Reverse, and Matching Series 











Subject Runs Trials Hits Av./25 Dev. 0 C.R. 

Mr. C.J.--direct 1000 25000 7387 7.39 2387 64.50 37.01 

Mr. C.J.--reverse 1000 25000 5126 5.13 126 64.50 1.95 

Matching Series 1000 25000 4911 4-91 - 89 64.50 - 1.38 
Table 4.2 


UT SERIES: DISTRIBUTION OF HIT FREQUENCIES 





Subject 01 8 5 4 5 6 7 


9 10 ll 12 15 14 15 16 





Mr. C.J.—direct O 3 14 22 50 100 159 166 176 154 78 48 21 


Mr. C.J.—reverse 5 27 65 122 165 209 170 125 


67 


55 15 


2 


1 


8 
.e) 


0 
ie) 


a 
0 0 





Table 4.4 


UT SERIES: FREQUENCIES FOR ISOLATED AND FOR CONSECUTIVE HITS 











Series 1 2 3 4 5 6 7 
Direct 3648 1034 345 82 45 8 5 
Reverse 3399 585 122 37 5 3 0 
Predicted 3264 627 120 23 4 1 0 

Table 4.5 
UT SERIES: 


Average Scores (per 25) for Successive Five Card Groups 








Series z II III Vv 
Direct 8.85 7.91 7.35 6.67 6.18 
Reverse 5.27 5.05 5.35 4.99 4.98 
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Table 4.3 


UT SERIES: 


Daily Flucutations in Scoring 





Table 4.5A 


UT SERIES: 


Hit Frequencies with 
Respect to Card Position 








Date Runs Direct Reverse 
Average Average 
Jan. 25 50 5.62 5.36 
Jan. 26 30 6.47 5.37 
Jan. 28 20 6.90 5.15 
Jan. 31 20 6.90 5.45 
Feb. 8 50 7.04 5.10 
Feb. 9 20 7.15 4.80 
Feb. 15 20 7.40 6.25 
Feb. 19 40 7.60 5.15 
Feb. 23 40 8.02 4-92 
Feb. 24 40 7.62 5.05 
Mar. 1 40 7.58 4-70 
Mar. 7 40 7.65 5.48 
Mar. 8 40 7.18 4-83 
Mar. 29 50 7.78 5.26 
Mar. “1 80 7.79 4.99 
Apr. 5 60 6.87 4.63 
Apr. 7 70 T4h 4.87 
Apr. 12 50 7.26 5.14 
Apr. 14 50 7.48 5.14 
Apr. 19 40 8.13 5.68 
Apr. 21 50 7.54 5.50 
Apr. 30 40 8.08 4.82 
May 3 60 7.66 5.23 














Card Direct Reverse 

Freq. Freq. 
1 356 209 
2 375 205 
3 373 200 
4 365 219 
5 300 221 
6 332 212 
7 333 189 
8 306 219 
9 314 179 
10 296 211 
11 325 208 
12 307 194 
13 283 283 
14 286 175 
15 268 209 
16 281 182 
17 290 201 
18 272 194 
19 244 219 
20 249 201 
21 244 193 
22 258 199 
23 264 209 
24 243 199 
25 226 196 
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Table 5.1 


EE SERIES: SUMMARY OF RESULTS 

























































































Direct Series Reverse Series 
No. of| Score| Av./25| Dev. o C.R. Score | Av./25| Dev. | C.R. 
Runs from from 
Dep. n.p. 
FE 400 2414 6.05 | 414 | 40.80} 10.15 1995 4.98 |} -5 | - .12 
EELR 120] 825} 5-88 1BS} BAIS] 5-08 665 | 4.785 | -37 [os 
FEo5_5 110 726 | 6.60 | 176 | 21.89] 8.22 548 | 4.98 | - 2 | - .09 
EEs 60 462 7.70 | 162 | 15.80] 10.25 315 5.25 | +15 | + .95 
DT (control) 140 611 5.79 | 111 | 24.135] 4.60 679 4.85 | -21 | - .87 
Total 710 4425 6.25 | 875 54.36} 16.10 3521 4.96 | -29 | - .55 
Total (Series)| 850 | 5256 | 6.16 | 986 | 59.48| 16.58 4200 | 4.94 | -50 | - .84 
Table 5.2 
EE SERIES: DISTRIBUTION OF HIT FREQUENCIES 
eo -«& 2 3 4 5 6 7 8 910 ll 12 15 14 
EER, Direct éeeécuaewn a 2S S&S €& 48 28 9 7 1 
Reverse ‘1.2 2 ses Fw @ 4. 26 Mm..6 %.8 
EER Direct aa a a oe ae oe i ee 
Reverse 1.77 3 Beak Ss & 7 ee 
EEg5_5 Direct 2. & a a ae oe oe CSE RS 8 
Reverse = 5S, BS 2B 6 & @ 8 A 
EE. Direct 0 Oo 0 1 3 5 8 9 185 2 a a 
Reverse 6 8 7 4 9 ll 4 14 8 = = 
DT (control ) Direct Oo 8 5 a: & tt 2 mB S&S 42 2 2 2 
Reverse e+ wD DB S2 17 12 9 "a 2 a oe 
* 
Total EE Direct 0 8 18 88 77 124 186 116 91 58 26 12 5 0O 
Reverse 1 15 56 108 185 186 107 88 89 21 8 2 41 0 
Total (Series) Direct 0 ll 21 46 101 #155 162 155 105 67 SO 15 § 1 
Reverse 1 22 66 123 160 168 124 100 48 25 10 8 1 O 
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Table 5.3A 
EE SERIES: 
Daily Fluctuations in Scoring 
Date Number Direct Reverse Date Number Direct Reverse 
of Runs Average Average of Runs Average Average 
EER, EEiR 
Sept. 29 40 7.28 5.13 Feb. 27 20 5.45 4.75 
Oct. 7 10 6.00 4-40 Mar. 1 20 6.35 5.15 
Oct. 17 10 4.50 5.20 Mar. 2 50 5.32 4.68 
Dec. 2 10 6.70 5.50 Mar. 3 20 5-90 4-55 
Jan. 6 30 6.43 4.20 Mar. 4 30 6.77 4.67 
Jan. 7 40 6.70 4-65 
Jan. 9 40 6.88 5.18 EE 5-5 
Jan. 11 20 5.25 4.65 
Jan. 13 30 5.30 5.00 Mar. 6 10 6.10 4-70 
Jan. 16 20 5.55 5.45 Mar. 30 40 6.43 4.87 
Jan. 18 10 6.20 4.60 Mar. 31 20 7-20 4-75 
Jan. 23 20 5.95 5.10 Apr. 3 40 6.60 5.27 
Jan. 24 20 5.55 5.55 
Jan. 25 40 5.52 5.22 EE, 
Jan. 26 10 4-50 5.10 
Jan. 30 30 5.40 4-63 Apr. 7 10 6.40 4.00 
Feb. 2 20 6.15 5.70 Apr. 11 10 6.90 6.20 
Apr. 12 10 7.20 4.00 
DT (control) Apr. 17 10 8.80 6.00 
Apr. 19 10 8.10 5.40 
Sept.28 20 6.35 3.65 Apr. 24 10 8.80 6.00 
Oct. 7 10 6.00 4.20 
Nov. 14 20 4-95 5.00 
Nov. 16 20 5.10 4-75 
Nov. 17 10 5.70 5.10 
Nov. 18 20 5.45 5.15 
Nov. 28 10 7.20 6.40 
Nov. 30 10 6.90 5.20 
Jan. 4 20 5.80 4-95 
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Frequency for Isolated and for Consecutive Hits 


Table 5.4 


EE SERIES: 


























Series 1 2 3 4 5 6 7 8 
EEnr, Direct 1361 304 107 27 2 y | 0 0 
Predicted 1306 251 48 9 2 0 0 0 

Reverse 1353 235 45 8 1 0 0 0 

EER Direct 495 103 20 9 4 1 0 0 
Predicted 457 88 27 3 1 0 0 0 

Reverse 434 83 12 3 3 0) 0 0 

EE25_5 Direct 359 85 37 6 4 , 23 
Predicted 359 68 13 3 0 0 0 0 

Reverse 388 55 13 ie) 1 1 0 0 

EE, Direct 249 68 19 5 0 0 0 0 
Predicted 196 38 7 1 0 0 0 0 

Reverse 206 36 7 4 fe) fe) fe) 0 

DT (control) Direct 522 95 23 6 e) a fe) 0 
Predicted 457 88 17 3 1 1?) 0 0 

Reverse 452 82 14 2 1 1?) i?) 1 

Total EE Direct 2464 560 183 47 10 + 0 0 
Predicted 2318 445 85 16 3 Q 0 0 

Reverse 2381 409 77 15 5 1 0 0 

Total Direct 2986 655 206 53 10 10 (6) 0 
(Series) Predicted 2774 533 102 20 4 1 0 0 
Reverse 2833 491 91 17 6 1 0 1 
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Table 5.5 


i EE SERIES: 


Average Scores (per 25) for Successive Five Card Groups 


























Method Direct Reverse 
I ot aR SS IV V I ps le IV v 
EEan 8.46 5.28 5.41 5.49 5-54 5.19 4-99 4-99 4.60 5.18 
EErR 8.50 5.00 5.07 5.25 5.57 4-96 4.68 4.89 4.39 4.75 
EE25_5 8.41 4.55 4-86 5.14 10.05 4-64 4.91 4-73 5.56 5.09 
EE 7.58 7.00 8.00 7.33 8.58 4-83 4-67 5.00 6.25 5.50 
Total 
EE 8.39 5.25 5.48 5.54 6.50 5.03 4.89 4.93 4.85 5.11 
DT 6.18 5.43 5.64 5.57 6.14 4-79 5.00 4.50 5.04 4.93 
Table 6.1 
MINOR VARIATIONS: SUMMARY OF RESULTS 
| Direct Series Reverse Series 
Method No. of Score Av./25 Dev. 0 C.R. Score Av./25 Dev. C.R. 
Runs from from 
n.p. n.p. 
DT, 110 899 8.17 349 21.39 16.31 522 4-74 -28 -1.31 
DTD 120 781 6.51 181 22.35 8.10 630 5.25 30 1.34 
DT op 70 511 7.30 161 17.06 9.43 367 5-24 17 1.00 
UTD 90 607 6.74 157 19.35 8.11 446 4-96 -4 - al 
UT 59 30 234 7.80 84 11.17 7.52 152 5.07 2 18 
UT a5 9 75 &. 33 30 6.12 4-90 40 4-44 ~>% - .82 





DT, and DT pp? 


Summary of Results for the Nine Pack Check 








Method No. of Runs Score Av./25 Dev. from n.p. C.R. 
DTy 990 4973 5.02 23 64.18 - 36 
DT rp 630 3149 4-99 - 1 51.20 -.02 
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MINOR VARIATIONS: 


Table 6.2 


DISTRIBUTION OF HIT FREQUENCIES 
































Experimental 
Technique Series 0 2 8 4 5 6 7 8 9 10 11 12 18 14 Runs Score 
Dt, Direct a oe ee eS & ee ee re oo ae ie ee ae eS 
Reverse 2 10 16 17 28 19 10 8 O 2 110.—s« 522 
Df Direct 0 a a & & FF S&S oe se ee ae ae eS he CS 
Reverse 0 TR NEO OG Se a Re. - 2 120 650 
Diep Direct 0 21 @ 6 © 2. Ss 6S Kee 70 = 3=«6§ll 
Reverse 0 ‘tm 4 Fae. See 70 867 
UTp Direct 0 a & C8 SR SE ae ee 90 607 
Reverse 1 2 €.H ae ae  B  S 90 446 
UT5o# Direct 0 oS @.f# 4. 8: 8 F 8-2 6s 50 6234 
Reverse 0O oe A 2-2 woe 50 6152 
UTy5* Direct 0 @. © 4 4.64 3 £2..:350 348 9 75 
Reverse 0 S42 te Be se g 40 
Total Direct 1 4 22 36 538 67 S86 68 55 85 25 18 2 6 429 8107 
Reverse 5 Sl 56 68 95 79 44 27 11 &§ 2 1 429 2157 





*Recorded frequencies pertain to arbitrary "runs" of 25. 
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Table 6.5 
MINOR VARIATIONS: AVERAGE SCORES (PER 25) FOR SUCCESSIVE FIVE CARD GROUPS 
Method Direct Reverse 
I II III IV Vv I II III IV v 
DT 10.59 8.23 8.36 7.32 6.36 4055 4050 5.05 4-36 5227 
DTp 7-96 5-54 5.83 5.50 7.71 4-71 5.58 6.08 5.42 4.46 
DTrp 7.64 7.29 7.57 7.21 6.79 5.86 5.57 4-86 5.14 4.79 
UTp 9.33 5.22 5.56 5.89 7.72 4-50 5.28 5.869 5.17 3.94 
UTs59 9.33 7.50 7.67 7.67 6.83 §.63 3.16 5.16 5.16 6.00 
UT75 10.55 7.22 8.33 9.44 6.11 4ehh 2.22 6.66 4.44 4ehh 
Table 7.1 
SUMMARY FOR ALL EXPERIMENTAL SERIES 
Direct Series Reverse Series 
No. of Score Av./25 Dev. o C.R. Score Av./25 Dev. C.R. 
Series Runs from from 
n.p. n.p. 
High Scorers 
Preliminary 1571 10556 6.59 2501 980.85 50.95 7861 65.00 6 07 
Main DT 2660 16478 6.19 5178 105.21 50.21 15349 «55.02 49 47 
Special Methods 2279 #15750 6.90 4555 97.59 45.41 11485 =5.04 88 -90 
Total High Scorers 6510 42564 6.54 10014 164.59 60.84 52695 5.02 145 .87 
Low Scorers 
Preliminary 1600 8267 5.17 267 81.60 5.27 7948 j$4.97 - 652 - .64 
Main DT 1520 7744 65.09 144 «#©79.55 1.81 7452 44.90 -168 -2.11 
Total Individual 
Subjects 5120 16011 5.15 411 115.95 53.61 15380 4.95 -220 -1.95 
Class 2840 14187 5.00 - 15 108.71 - .12 14049 4,95 -151 -1.59 
Total Low Scorers 5960 50198 5.07 5398 157.49 2.52 29429 44.94 -571 _2.56 
Grand Total 12470 72762 5.85 10412 227.81 45.70 62122 4.98 -228 -1.00 
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Table 6.6 


DTqp! HITS DERIVED FROM THE MATCHING OF CALLS AGAINST TARGET AND NON-TARGET PACKS 











Pack Total unin- * 
tended hits Ti 
Call* 1 2 5 4 5 6 7 8 9 10 percall  av./25 ¥a 
Rat 
+ 
1 9 ss 4«& SS Sse 60 61 60 60 75 #£517** 5.92 if 
2 oa 2 nr 0 © @ @ 8 Ff 2 546 5.51 it 
5 “oe 6 8#® a & Bo 468 4.75 oH 
4 57 s &@ # a2 2 CR 52 51 509 5.14 me 
5 ees st 8 @#¢ 8&8 @ @ 508 5.15 qs 
6 a ss ss 2.4 2 2B ee eS 497 5.02 - 
7 So a. €¢ « 86 424: 8 82 oe 478 4.83 E 
- a 2 8 2 8 @  &. £2 2S 475 4.80 “I 
4 eo es sa OO 8 2. 242s & 474 4.79 | 
10 ss 8S S86 6 6 46 s6 Sl 8 8 501 5.06 : 
Total unintended 
hits per pack 481" 471 507 516 479 492 506 512 492 517 4975 
(diagonal cells 
omitted) 
Average per 25 4.86 4.76 5.12 5.21 4.864 4.97 5.11 5.17 4.97 5.22 5.00 





* Call l: Call intended for Pack l. 
*# Cell expectancy: 55; column expectancy 495. Direct Av: 7.30 


DTy: HITS DERIVED FROM THE MATCHING OF CALLS AGAINST TARGET AND NON-TARGET PACKS 








Pack Total unin- 
tended hits 
Call* 1 2 3 4 5 6 7 8 9 10 percall  av./25 
1 oF sea 45 589 #=+%SO 87 #=%SO 59 88 #50 520 ** 5.08 
2 a Ss. so .84 a @ 2 Be. eae 825 5.16 
8 4 4 46 «SO 4 #86 4 38 389 29 538 5.87 
4 ss Bp sf &# a 2 2S ewe US 308 4.89 
5 ”. RP. 2... oS. 2 OR. we ae Se 290 4.60 
5 se Ss 2 2B ea SS Se OS ee eS 537 5.55 
7 se 88lCiSKA“sé‘‘:COKCOHCOCiaDsiaDDtséS: 296 4.70 
8 sf rm a Se SS. ee eS 315 5.00 
a Sl 686i Sst sCi si‘ ‘S“SCé<‘<C TCC S12 4.97 
10 ‘A 8. 2.82. 38 @ ©€6 Bae Se 505 4.84 





Average per 25 4.98 4.98 5.05 4.98 4.77 5.11 5.45 4.95 4.75 4.97 4.99 





* call 1: Call intended for Pack 1. 


* Cell expectancy: 55; column (or row) expectancy: 515. 
Direct Av: 8.17 











Table 7.2 


SUMMARY FOR ALL EXPERIMENTAL SERIES: DISTRIBUTION OF HIT FREQUENCIES 





High-Scoring Subjects 












































0 ss 3 4 5 6 7 3 9 20 6 An 4.28 28 14 26. 28 17 
Preliminary Direct 4: 2. a Se Oe. - ee eee ee. ee ae a eS Ue US fe UR 
Reverse 6 SM 121 217 ate 3s até 1a ll COU 8 2. 6.3% 2 £ 8 
PS Main DT Direct : 2 2-2. Pe De Ae a. Se a ei aC LD 
S Reverse 14 57 202 330 533 491 426 295 1863 87 2 15 oe A ae 
= Special Direct re : 17 - 2 2 | 3-3 2K 2 a OM A A 
5 Methods Reverse 7 57 160 301 391 470 373 269 1462 «#71~«228 7 i. £2 & of SS 
— 
wa Total Direct 7 67 274 200 G2. O94 1055. 1695 600 594 334 202 % 45 22 6 2 1,1 
- Reverse 27 150 483 848 1203 1263 1065 729 428 205 69 30 OR Vee ee a ee 
fa 
< Low-Scoring Subjects 
Ay 
fx4 Preliminary Direct i we ee! ee ee. a. i) | a: a | 4 ae ie | 
— Seeetee: 7. 39 19k. 200: 296 307? 24k 104 214-48 < 0 7 a @& «6 
a Main DT Direct 7 9: of S63 26 3. Bas 2 89 62 20 5 , &. 4% 
= Reverse 7 38 103 230 302 302 236 156 82 38 16 6 4 fe) 0 
— Total Direct 11 73 190 419 527 638 480 400 198 124 46 9 .. & 2 
pend (Individual) Reverse 14 77 227 469 578 609 477 340 196 80 34 13 Se a 
Class Diveet: 15 70 190 404. 326 $72 438: -395- 166 ©. 32. eS ae 
Reverse 17 69 200 414 532 562 421 #314 186 80 30 12 3.0 0 
Total Direct 26 143 380 823 1055 1210 895 735 366 214 78 23 & f A 
Reverse 31 146 427 4883 1110 1171 898 654 382 160 64 25 ¢ @ *® 
All Subjects 
Grand Direct 33 210 554 1223 1753 2104 1950 1770 1246 808 412 225 100 47 25 6 2 1 1 
Total Reverse 58 296 910 1731 2313 2434 1963 1383 810 365 133 55 17 #1 #O 1 
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se a * eas 
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Table 7.3A 
SUMMARY FOR ALL EXPERIMENTAL SERIES: DAILY FLUCTUATIONS IN SCORING 
Wov. 4 60 5.18 4-66 May 12 60 6.12 5.12 
Nov. 5 80 5.78 5-06 May 23 30 6.80 4.63 
Nov. 9 90 6.18 468 May 24 20 6.20 5.65 
Nov. 10 100 7.04 4-56 May 25 40. 5.90 5.10 
Nov. 11 100 7.49 5.03 Sept. 28 40 5.40 4.07 
Nov. 12 60 5.95 4.76 Sept. 29 40 7.28 5.13 
Nov. 16 90 7.39 4.98 Sept. 30 20 5.50 4.90 
Nov. 18 90 6.89 4-84 Oct. 3 10 4-70 5.60 
Nov. 19 10 4.30 5.40 Oct. 5 40 5.10 4.68 
Nov. 22 30 5.77 5-10 Oct. 7 20 5.35 4.85 
Nov. 23 70 7.30 5424 Oct. 8 25 5.52 5.04 
Nov. 24 30 6.20 5.10 Oct. 11 85 5.02 4.65 
Dec. 1 30 7.83 5.00 Oct. 12 15 5-20 4.20 
Dec. 2 30 6.83 4.97 Oct. 13 20 5.25 5.20 
Dec. 3 30 4-73 55443 Oct. 17 10 4.50 5.40 
Dec. 5 30 4-90 4.70 Oct. 19 25 4-88 5.08 
Dec. 5 20 5640 4435 Oct. 25 450 4-79 5.08 
Dec. 13 10 5430 4.80 Oct. 26 10 5.80 4.50 
Oct. 27 550 4.97 4.98 
Jan. 5 40 6.17 5.42 Oct. 28 30 4.63 5.40 
Jan. 25 50 5.62 5.36 Nov. 1 20 5.20 4.50 
Jan. 26 30 6.46 5.36 Nov. 4 30 4-86 4.13 
Jan. 28 20 6.90 5.10 Nov. 6 20 4.80 5.25 
Jan. 31 20 6.90 5.45 Nov. 11 30 513 4.60 
Feb. 3 10 5.80 4.60 Nov. 14 20 4-95 5.00 
Feb. 8 50 7.04 5.10 Nov. 15 20 5.30 4.85 
Feb. 9 20 7.15 4-80 Nov. 16 26 4.85 4.65 
Feb. 15 20 7.40 6.25 Nov. 17 10 5.70 5.10 
Feb. 19 40 7.60 5.15 Nov. 18 20 5445 5.15 
Feb. 23 50 7.58 4.98 Nov. 20 20 4-85 5.20 
Feb. 24 40 7.62 5.05 Nov. 28 10 7.20 6.40 
Mar. 1 50 7.22 5.00 Nov. 30 10 6.90 5.20 
Mar. 7 40 7.65 5447 Dec. 2 10 6.70 5.50 
Mar. 8 100 6.16 5.09 Dec. 6 10 4.80 5.40 
Mar. 29 50 7.78 5.26 
Mar. 31 80 7.78 4.98 Jan. 10 50 5.60 4.44 
Apr. 5 64 6.75 4075 Feb. 4£ 90 5.430 5.30 
Apr. 7 70 Te4h 4-87 Feb. 5 20 4.30 4.85 
Apr. 12 50 7.26 5.14 Feb. 6 10 7.30 5.40 
Apr. 14 50 7.48 5.14 Feb, 8 10 5.90 4.60 
Apr. 19 530 5.29 4-93 Feb. 9 30 5.20 443 
} Apr. 20 620 4.93 5.00 Feb. 10 40 5.52 4.65 
Apr. 21 50 7.54 5.50 Feb. 12 9 6.11 5.44 
Apr. 22 800 5.04 4.81 Feb. 16 10 7.40 4.00 
Apr. 26 20 5.05 4.95 Feb. 17 90 6.28 4.81 
Apr. 27 10 5.10 5.00 Feb. 18 70 5.54 4-94 
Apr. 28 60 5.47 4492 Feb. 19 66 4.84 4.90 
Apr. 29 30 6.16 5.00 Feb. 20 20 6.00 5.45 
Apr. 30 40 8.07 4.82 Feb. 21 40 4.85 5.22 . 
May 2 10 4.70 4.00 Feb. 23 94 5619 4-81 - 
May 3 60 7.66 5.23 Feb. 25 10 4-50 5.60 4 
May = 5 30 6.60 4.93 Mar. 1 10 5.50 4.80 a 
May 9 20 6.15 4.65 Mar. 2 50 5.20 5.08 4 
May 10 10 7.50 5.30 Mar. 3 30 5.63 5.40 4 
May ll 20 6.70 4.75 Mar. 5 30 5.10 4.86 ¥ 
Mar. 6 30 6.83 4.96 a 
5 
a 
a 
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Sept. 
Oct. 
Oct. 


30 
1 
5 


130 


. wae aw & hme 
oooorr owned 


SBSSSsReEvnRERSsSSssnsRrsss 


. . i . 
rPOnOL WU @ 


AMON wMn ae WDWAUEU*eurwnwnwn 


“ 
w 
wr 


5.00 
6.00 
6.05 
5.73 
5.53 
8.63 
5.40 
5.95 
5.40 
5.76 
5.40 
5.60 
4-40 
5.96 


Table 7.3A (cont'd) 


5.84 
5.30 
4-61 
5.30 
6.15 
425 
4.87 
4-73 
5-40 
4.18 
4-80 
4-69 
4.85 
4-90 
5.15 
5.07 
5.33 
5.05 
5.20 
4-97 
5.12 
4-75 
4-71 
5.00 
5.08 


> 

ee 
~ 
w 


no 


OBRAKHOORIHOBHPEHINODOAONA 


MPO UU! US UU S| MU TS 
SSSEBSRSAEFRBSSESSSSSSRAESSE 


5.05 
5.60 
4.00 


4-66 
4-90 
5.00 


4.80 
4.48 





Oct. 6 SO 5.76 4.24 
Oct. 7 120 5.74 5.92 
Oct. 8 40 6.17 4.95 
Oct. 11 70 4.83 4.83 
Oct. 12 140 6.64 5.38 
Oct. 13 70 6.15 5.08 
Oct. 14 80 7.47 4.97 
Oct. 15 70 6.05 4.94 
Oct. 18 80 6.65 5.18 
Oct. 19 130 6.44 5.15 
Oct. 20 70 6.45 5.28 
Oct. 21 140 7.14 5.12 
Oct. 25 130 5.88 4.76 
Oct. 26 10 5.00 4.60 
Oct. 27 80 5.58 5.02 
Oct. 28 160 5.67 4-91 
Oct. 29 50 Thdh 5.24 
Nov. } | 50 5.40 4.80 
Jan. 4 20 5.80 4.95 
Jan. 6 30 6.43 4-20 
Jan. 7 40 6.70 4.465 
Jan. 9 40 6.87 5.17 
Jan. 11 20 5.25 465 
Jan. 13 30 5.20 4-86 
Jan. 16 20 5-55 5-45 
Jan. 18 10 6.20 4.60 
Jan. 23 20 5.95 5.10 
Jan. 24 50 4.86 5.38 
Jan. 25 40 5.52 5.22 
Jan. 26 30 5.03 5.30 
Jan. 30 30 5.40 4.63 
Feb. 1 30 5.20 4.53 
Feb. 2 20 6.15 5.70 
Feb. 6 40 5.97 5.52 
Feb. 10 30 «65.10 865.40 
Feb. 27 20 545 4.75 
Mar. 1 20 6.35 5.15 
Mar. 2 50 5.32 4.66 
Mar. 3 20 5-90 4.55 
Mar. 4 30 6.76 4.63 
Mar. 6 10 6.10 4-70 
Mar. 30 40 6.42 4.87 
Mar. 31 20 7.20 4.75 
Apr. 3 40 6.60 5.27 
Apr. 7 10 6.40 4.00 
Apr. 11 10 6.90 6.20 
Apr. 12 10 7.20 4.00 
Apr. 17 10 8.80 6.00 
Apr. 19 10 8.10 6.40 
Apr. 24 10 8.80 6.00 
May 17 39 7.92 492 
June a 20 5.20 5.20 
June 6 20 5.05 4-40 
June 7 10 4.70 4.60 
June 16 20 5.80 5.30 
June 19 40 4.70 4.85 
Aug. 10 40 5.25 4.90 
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Table 7.4 


SUMMARY OF ALL EXPERIMENTAL SERIZS: FREQUENCIES FOR ISOLATED AND FOR 
CONSECUTIVE HITS 


Low-Scoring Subjects 



































1 2 3 4 5 6 7 8 > 2 ae ae 
Direct 5320 1056 207 42 ae ae 
Preliminary Predicted 5222 1004 193 37 _ ee Sn 
Reverse 5130 983 200 47 , Ree 1 
Direct 5040 911 216 33 , 63 
Main DT Predicted 4961 953 183 35 : ee ae 
Reverse 5048 850 170 34 - Pa Bene | 
Total Direct 10960 1967. ‘423... 75 ..27 9.3 
Individual Predicted 10184 1957 376 72 14 2 1 
Subjects Reverse ie «6aes. 3.. B-.2a4 22-2 
Direct 9341 1705 331 90 13 3 18] 
Class Predicted 9270 1781 342 65 13 2 1¢) 
Reverse Gi9s 2798... 395:.. 15 4:8 
Direct 19701 3672 754 165 30 10 3 
Grand Total Predicted 19454 3738 717 137 26 5 1 
Reverse 19973 330 TS te 6S 
High-Scoring Subjects 
Direct $94. i9TR., 4a: COR SD Be. OE Oe 8 
Preliminary Predicted 5128 985 189 36 . £2 £2.44 2s 
Reverse $167 939 213 3% 3) ££ 2.8 ee Bee 
Direct 9207 2251 542 152 64 28 5 0 0 0 1?) 1 
Main DT Predicted 8682 1668 320 61 12 2 i¢) i¢) ie) 0 0 0 
Reverse 8880 1570 337 58 a Fae te ae ay ee 
All remain- Direct 8110 2121 694 183 7h 23 8 0 1¢) 1?) 0 0 
ing series Predicted 7439 1429 274 53 10 20000 0 90 
with Mr.C.J. Reverse 7712 1367 252 53 11 1 0 1 ¢) QO ie) 0 
Direct 22901 5744 1660 433 186 59 18 4 1 0 ie) 1 
Grand Total Predicted 21249 4083 783 150 29 6 1 00 0 0 0 
Reverse 23799 3% GR i 22:0 2 8: *& 2 ee 
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Table 7.5 


SL 


SUMMARY FOR ALL EXPERIMENTAL SERIES: 


Average Score (per 25) for Successive Five Card Groups 








Subjects Direct Reverse 
I II III IV Vv I II III IV V 
Low Scorers 5.16 5.10 5.02 5.17 5.21 4-96 4.84 4.92 4.98 4.85 
High Scorers 

(minus C.J.?) 6.10 6.14 6.20 6.20 6.06 5-10 4.95 5.12 4.98 4.91 
Total 

(minus C.J.) 5-61 5.60 5.58 5.66 5.61 5.03 4-89 5.01 4.98 4.93 
Mr. C.J. 

(minus EE) 8.00 7.18 6.98 6.43 6.35 4-97 5-1l 5.24 5.09 4.84 

Mr. C.J. EE 8.39 5.25 5-48 5.54 6.50 5.03 4-89 4.93 4.85 5.11 
Individual 

Total 6.50 6.05 6.52 5.88 5.90 5.01 4.96 5.08 5.00 4.92 

Class 4-97 4094 5.06 4694 5-07 4-87 4.94 4.92 5.00 5.00 

Grand Total 6.18 5.80 5.79° 5.67 5.71 4-98 4.95 5.04 5.00 4.94 
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Table 7.5A 
SUMMARY FOR ALL EXPERIMENTAL SERIES: 
Hit Frequency with Respect to Card Position (Direct) 
Card Low High Total Mr. C.J. Mr. C.J. Indi- Class Grand 
Scorers Scorers minus (minus EE) EE vidual Total 
Mr. C.J. Total 

1 696 738 1434 950 251 2635 583 3218 
2 657 705 1362 985 283 2630 584 3214 
3 596 642 1238 978 237 2453 525 2978 
4 629 678 1307 920 250 2477 602 3079 
5 640 714 1354 821 170 2345 527 2872 
6 606 681 1287 873 155 2315 551 2866 
7 665 727 1392 816 153 2361 608 2969 
B 633 685 1318 825 142 2285 544 2829 
9 626 719 1345 827 161 2333 558 2891 
10 655 690 1345 835 135 2315 547 2862 
11 600 700 1300 844 161 2305 595 2900 
12 611 740 1351 844 160 2355 551 2906 
13 641 699 1340 810 143 2293 578 2871 
u 674 701 1375 762 148 2285 566 2851 
15 607 697 1304 802 166 2272 586 2858 
16 628 681 1309 772 145 2226 584 2810 
17 649 727 1376 772 149 2297 567 2864 
18 664 723 1387 746 157 2290 548 2838 
19 635 677 1312 708 167 2187 568 2755 
20 647 726 1373 743 169 2285 537 2822 
21 665 739 1404 702 183 2289 543 2832 
22 671 705 1376 766 177 2319 593 2912 
23 671 673 1344 755 190 2289 548 2837 
24 604 627 1231 756 188 2175 581 2756 
25 641 709 1350 715 185 2250 613 2863 








5 Ti teesolatgct che MN Be (28 


IST Sheree y 


et Sa or 





248 





JOURNAL OF PARAPSYCHOLOGY 





SUMMARY FOR ALL EXPERIMENTAL SERIES: 


Table 7.5A 


Hit Frequency with, Respect to Card Position (Reverse) 








Card Low High Total Mr. C.J. Mr. C.J. Indi- Class Grand 
Scorers Scorers minus (minus EE) EE vidual Total 

Mr. C.J. Total 
1 588 608 1196 551 146 1893 538 2431 
2 635 590 1225 601 136 1962 557 2519 
3 606 596 1202 581 138 1921 586 2507 
4 643 546 1189 574 153 1916 535 2451 
5 626 568 1194 583 141 1918 548 2466 
6 619 573 1192 601 140 1933 559 2492 
7 625 547 1172 569 140 1881 576 2457 
8 585 610 1195 624 139 1958 564 2522 
9 594 544 1138 582 138 1858 521 2379 
10 594 550 1144 597 137 1878 586 2464 
11 616 586 1202 581 141 1924 553 2477 
12 599 532 1131 565 121 1817 544 2361 
13 641 699 1340 810 143 2293 580 2873 
14 595 551 1146 525 154 1825 557 2382 
15 618 551 1169 569 141 1879 561 2440 
16 632 599 1231 587 147 1965 535 2500 
17 613 525 1139 590 132 1860 558 2418 
18 653 578 1231 578 135 1944 602 2546 
19 599 569 1168 599 138 1905 571 2476 
20 612 566 1178 606 136 1920 576 2496 
21 620 558 1178 540 142 1860 554 2414 
22 631 541 1172 577 136 1885 624 2509 
23 536 584 1220 570 166 1956 544 2500 
24 623 547 1170 567 130 1867 567 2434 
25 577 570 1147 565 151 1863 553 2416 























VARIATIONS OF TIME INTERVALS 
IN PRE-SHUFFLE CARD-CALLING TESTS* 


Lois Hutchinson 
Parapsychology Laboratory, Duke University 


INTRODUCTION 


History of the Problem. The hypothesis of pre- 
cognitive ESP states that future events of a random 
character can be identified to some degree by means of 
ESP. Historically, this hypothesis originated outside 
the psychological laboratory. The literature of the 
societies for psychical research, of which the report of 
Saltmarsh+ is typical, contains many accounts of dreams 
and waking spontaneous experiences of presumably precog- 
nitive character. But the problem of precognition is 
also one which arises inferentially from the results of 
laboratory studies of ESP. 





As a matter of fact, the problem of the occur- 
rence of precognition may clearly be subjected to experi- 
mental test, and the results of some laboratory studies 
have already been reported. As a first step toward test- 
ing the hypothesis, Rhine~ had subjects make calls for a 
to-be-determined sequence of events that, so far as 
could be judged, could not be perceived sensorially, in- 
ferred rationally, nor brought about by the subject de- 
liberately. The tests, "pre-shuffle card-calling," were 
made with ESP cards; the subject was asked to predict 
the order a pack would have after it had been shuffled 
by the experimenter. These tests yielded results which 
deviated significantly from chance expectation, but the 
interpretation of their bearing upon the precognition 





* A thesis submitted in partial fulfillment of the requirements 
for the degree of Master of Arts in the Graduate School of Arts 
and Sciences of Duke University (1940). The writer is indebted 
to Dr. J. B. Rhine for his guidance of the research, and to Dr. 
J. G. Pratt for his many helpful suggestions and painstaking 
supervision of this study. The assistance of other persons in 
the experiments is mentioned in the text and is gratefully ac- 
knowledged. 

Saltmarsh, Hi. F., "Report on Cases of Apparent Precognition, " 
Proceedings of the S. P. R., 1934, Vol. XLII, p. 49. 

Rhine, J. B., "Experiments Bearing on the Precognition Ilypothe- 
sis," Journal of Parapsychology, March, 1938, Vol. II, p. %8. 
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hypothesis has since been qualified by the results of 
Antes tests, the "ESP shuffle." Rhine, Smith, and Wood- 
ruff? reported that subjects could apply ESP to the task 
of shuffling a pack of cards. This suggests that the 
person shuffling the cards in the pre-shuffle card-call- 
ing tests might have used ESP to effect better-than- 
chance scores, an explanation which would make the pre- 
cognition hypothesis unnecessary for the interpretation 
of the results. The ESP shuffle work emphasized the in- 
portance of using an impersonal procedure to "randomize" 
the cards in any adequate test of precognition,. 


Two other laboratory tests concerning the hypo- 
thesis of precognition are those of Carington*, who 
asked his subjects to predict the faces_that would turn 
up on the throw of dice, and of Tyrrell?, who used an 
electric ESP machine. In Tyrrell's experiment the sub- 
ject was asked to open one of five boxes which would be 
mechanically selected to be lighted by an electric bulb 
following the response of the subject. Although both 
experimenters state that their results are relevant to 
the problem of precognition, these results cannot be con- 
sidered as conclusive. Carington states, "It is clear 
that these 40 subjects taken as a group, afford no justi- 
fication for supposing that any precognitive faculty is 
at work, for the total number of successes is exactly 
equal to expectation." His conclusion that his results 
are significantly above expectation is made on the basis 
solely of a selection of individual subjects who scored 
above and below chance expectation. Such procedure is 
subject to the counter-hypothesis of selection of data, 
and the results must be discounted accordingly. fTyrrell's 
results, while indicating ESP, are not necessarily in- 
terpreted as of precognitive character betause (1) the 
adequacy of the random character of the mechanical se- 
lector has not for the series in question been fully de- 
termined, and (2) the results could be interpreted as the 
outcome of the subject's clairvoyant knowledge of the 
mechanical selector's direction of choice. 





3, Rhine, J. B., Smith, Burke M., and Woodruff, J. L., "Experiments 
Bearing on the Precognition Hypothesis: II The Role of ESP in 
the Shuffling of Cards," Journal of Parapsychology, June, 19%, 
Vol. II, p. 119. 

4. Carington, Whately, "Preliminary Experiments in Precognitive 
Guessing," Journal of the Society for Psychical Research, June, 
1935, Vol. XXIX, p. 86. 

5. Smith, Burke M., "The Tyrrell Experiments," Journal of Para- 
psychology, March, 1937, Vol. I, p. 63. 
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Scope of the Research. In the experiments pre- 
sented here, the subjects were asked to call decks of 
ESP cards as they would be arranged at some future time, 
with random rearrangement intervening. A mechanical 
shuffler, which has been described in detail by Smith®, 
was rotated a fixed number of times to determine the ran- 
dom arrangement of the cards. 





Two time intervals are contrasted: (1) calling 
for checking with the random rearrangement taking place 
within twenty-four hours, (2) calling for delayed check- 
ing, with random rearrangement taking place ten days af- 
ter the calls are made. 


In a first phase of the investigation, to be de- 
signated as Series I, the subjects were not told any of 
their scores until the last results were checked, twenty 
days after the experiment began. In the second phase, 
Series II, the subjects were told their scores daily for 
the ONE-DAY check group, but did not, of course, learn 
their scores on the TEN-DAY check series until at least 
ten days later. One aspect of the investigation will be 
concerned with the relative effects of telling the sub- 
ject his scores at these two intervals after the time of 
making the calls. 


Statement of Problems. This report will be con- 
cerned with three problems: (1) Can calling the order 
of cards with random rearrangement intervening be demon- 
strated by normal subjects to an extent greater than 
chance expectation? (2) If so, how does success at call- 
ing the order of cards one day ahead compare with calling 
the order for ten days ahead? (3) If these two intervals 
of time affect the scores differently, how is this dif- 
ference to be explained? 





PROCEDURE 


General Statement. The experimental procedure 
involved the use of the text known as "precognitive down 
through," or PDT, in which the subject predicts the order 
of cards in ESP decks as they will be after they are 
shuffled some time after the responses are made. 





There were in reality two experiments made with 
this general procedure. The two experiments will be de- 
Signated, in the order in which they were performed, as 





6. Smith, Burke M., "Note on a Shuffling Machine," Journal of Para- 
psychology, September, 1938, Vol II, p. 231. 
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Series I and Series II, respectively. As Series II was 
planned as a separate experiment involving certain con- 
ditions different from those of Series I, it will be 
necessa to describe the two procedures and to consider 
the results of each part separately. The necessity for 
this procedure will become evident as the plan of the in- 
vestigation is explained. 


Procedure for Series I. Sixteen subjects, all 
graduate women students residing in Faculty Apartments, 
took part in Series I. 





In preparation for the experiment, the investi- 
gator first wrote out a description of the purpose and 
plan of the experiment, stating exactly the number of 
days the experiment would run and estimating that be- 
tween fifteen and twenty subjects would be used. This 
prefatory statement was filed with the record librarian 
of the Parapsychology Laboratory, who then issued to the 
experimenter the number of record sheets which would be 
required. These sheets were numbered serially and each 
one was stamped with the seal of the Laboratory, which 
was accessible only to the librarian. Record of the 
serial numbers and of their intended use was kept, and 
no other similar sheets have been issued with any of the 
same numbers. The serial numbers therefore served as a 
convenient means of identifying the data sheets at all 
times, and the degree to which the numbers are all re- 
presented among the "raw data" sheets of the completed 
experiment is an objective criterion for judging the 
likelihood of the selection of data. 


At the beginning of the series, the experimenter 
saw each of the subjects personally, gave to each one 
ten of the record blanks, and told her the purpose of 
the experiment and the procedure to be followed in making 
the calls. Written instructions were not read, but the 
investigator was careful that each subject understood 
what she was to do and the procedure that was to be fol- 
lowed. Each subject was told that sometime on the even- 
ing of Februa 26, and on each evening thereafter for 
ten days in all, she was to predict in the ten CALL 
columns of the appropriate record sheet the order of ten 
decks of ESP cards as they would be after being shuffled 
later. The correct order of cards would be recorded in 
the adjacent CARD column at the time appointed for check- 
ing the scores; if the subject preferred, she could take 
as her task to predict for each space in the CALL column 
what would later be entered in the CARD space just to 
the right. The subject was told that the first five 
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runs on each sheet would be filled in and the scores 
checked within twenty-four hours of the time of making 
the calls, while the cards for the second five runs 
would not be shuffled until ten days had elapsed. In 
order to make all conditions except the varied time in- 
ternals as similar as possible, the experimenter asked 
the subjects to fill out the first column on the sheet, 
then the sixth--which was, of course, the first for the 
tenth-day check--the second, then the seventh, etc. 


The subjects were advised that they would re- 
ceive for the two conditions similar cash rewards for 
high scores, according to the following schedule: 


$ .05 for a score of 8 
" " " " 


-10 9 
3 25 " " " " 10 
‘ 50 " " " " 1 1 
1 - 0 0 " " " " 1 2 
9 ‘ 50 " " " " 13 
4 , 0 0 " " " u 14 


They were also informed that they would not be told any 
of their results until the experiment was completed and 
checked, that is, at the end of the twentieth day. 


The experimenter collected the record sheets 
from the subjects each morning and turned them over to 
Mr. E. P. Gibson, who had charge of shuffling the cards. 
Five separate decks of ESP cards were shuffled on the 
day following the making of the calls by placing each 
in the shuffling machine which was rotated exactly three 
times. The decks were recorded with the cards in the 
order in which they came from the shuffler, the first 
deck shuffled as run number 1, the second as number 2, 
and so on. All the subjects' responses were checked 
against the same set of five decks. The cards were re- 
corded in the first five CARD columns of each record 
sheet by Mr. Gibson, using a gelatin duplicator. No 
sheets which were not stamped at the proper time by the 
librarian are counted in the totals. 


The librarian then gave the stamped sheets for 
each day to Miss Maxine Slaughter , who without marking 
the sheets in any way, counted the number of hits by 
comparing the entries under the adjacent CALL and CARD 
columns for the first run, then for the second, and so 
forth. The experimenter also checked the number of hits, 
likewise without marking the data sheets in any way, and 





* Graduate student in the Department of Psychology, Duke University. 
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the two records were kept separately. At the conclusion 
of the experiments, all the rgcords received an independ- 
ent check by Miss Esther Bond or Miss Maxine Slaughter, 
and any discrepancies between their scores and those of 
the experimenter were resolved by carefully consulting 
the record sheet. Discrepancies were found to have 
arisen predominantly through overlooking hits, a finding 
in keeping with the experience of Greenwood‘ in checking 
and re-checking 20,000 runs. The record sheets of this 
experiment, still unmarked, are open to further independ- 
ent check. 


At the proper interval for the tenth-day check, 
the record sheets were again given to the librarian and 
decks for the second five runs of each sheet were shuf- 
fled in the machine as already described and the cards 
were recorded on each sheet by means of the duplicator, 
The results were checked and independently re-checked as 
for the ONE-DAY check series. 


All computations entering into this report were 
made twice. Miss Esther Bond and the experimenter made 
independent computations, and in case of discrepancies 
the work was re-checked until complete agreement was ob- 
tained. 


Procedure for Series II. Twenty-four subjects 
(9 graduate women students, 5 graduate men students, one 
undergraduate woman, 2 undergraduate men, one college in- 
structor, one school teacher, one housewife, one mechan- 
ical engineer, and 3 secretaries) took part in Series Il, 
The procedure and conditions were the same as in Series I, 
with the following exceptions: 





(1) Instead of making their predictions on ten 
successive days as previously, the periods of making the 
calls were divided into two groups of five days each, 
the groups being separated by an interval of three days' 
recess. These two parts of the experiment will be de- 
signated [I-A and II-B, respectively. Series II-A, be- 
ginning on the evening of April 9, involved eighteen of 
the above twenty-four subjects. In Series II-B, which 
started on April 17, twelve subjects continued from II-A 
and six additional ones joined the experiment. 


(2) Instead of waiting until the last results 
were checked to hear of their scores, the subjects were 
told that they would be informed from day to day what 





7, Greenwood, J. A., "Analysis of a Large Chance Control Series of 
ESP Data," Journal of Parapsychology, June, 1938, Vol. II, p. 14h 
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scores they made on the first half of each sheet, that 
is, the half for the ONE-DAY check. They could not, of 
course, learn their scores on the TEN-DAY check series 
until ten days after the calls were made. 


(3) The subjects were not specifically told in 
what order to fill in the CALL columns of their record 
sheets. Five of the subjects who had taken part in 
Series I are known to have followed the staggered order 
used then, but it may be supposed that most of the 
sheets were filled out in regular order from left to 
right. That is, the five ONE-DAY check runs were re- 
corded in the first half of each period and the five 
TEN-DAY check runs in the second half. The possible 
effect of this order of making the calls will be taken 
into account in discussing the probable significance of 
the difference in results between ONE-DAY and TEN-DAY 
checking. 


RESULTS 


Results of Series I (Knowledge of scores with- 
held until end of experiment). There were 1430 runs 
made in Series I. Of this number, 715 were checked with- 
in the same twenty-four hour period and are grouped as 
ONE-DAY check, while 715 were checked ten days after the 
sheets were filled out and are grouped as TEN-DAY check. 
The total deviation for the 715 runs of the ONE-DAY 
check group is -4; the total deviation for the 715 runs 
of the TEN-DAY check group is +28. The results of both 
groups are not significantly different from chance ex- 
pectation; nor is there any unusual difference between 
the ONE-DAY and the TEN-DAY checked groups (See Table 1). 





Although the results of Series I were inconclu- 
Sive, one factor of interest emerged. A comparison of 
the total deviations of each subject for the ONE-DAY 
checked runs with her total deviation for the TEN-DAY 
checked runs indicated that 13 pairs of the 16 deviation 
Signs were identical. That is, if a subject had a nega- 
tive deviation for 50 runs on the ONE-DAY checked series, 
more often than not her deviation for the 50 runs on the 
TEN-DAY checked series would also be negative. A bi- 
nomial evaluation of this result in the experiment gives 
&® probability value of .01, suggesting that the identity 
of direction of the 13 out of 16 pairs of deviations in 
& group of unselected subjects may not have been wholly 
due to chance (see Table 1). In any event, an evalua- 
tion based on so few data would be suggestive rather 
than conclusive. 
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TABLE 1 


Results of Series I showing the subjects used, number of 
runs of each subject, deviations from chance expectation, 
and totals. 



































ONE-DAY Check TEN-DAY Check 
Series Series 

Subjects Runs Deviations Runs Deviations 
C.B. 50 +21 50 - 8 
E.B. 15 a | 15 +10 
H.B. 45 - 8 45 -17 
A. F. 50 -14 50 +10 
MH. 50 + 3 50 - 1 
I.M. 50 + 1 50 +11 
N.M. 50 -12 50 - 3 
H.R. 50 + 4 50 +23 
J.R. 30 -21 30 -16 
M.S. 45 +18 45 +12 
E.S. 40 -12 40 - 1 
L.T. 40 + 7 40 +12 
E.T. 50 +19 50 +18 
G.V. 50 -13 50 -13 
H.W. 50 +12 50 + 5 
B.W. 50 -11 50 -14 
Total 715 - 4 715 +28 
S.D.: 715 runs = 53.48 S.D.: 715 runs = 53.48 
C.R. = 0.07 C.R. = 0.52 

S.D.: 1430 runs = 75.63 

C.R. = 0.42 


Results of Series II. In Series II the total 
number of runs that were checked within the same twenty- 
four hour period in which the subjects filled out the 
sheets, equals 835 and has a positive deviation of 180. 
The average number of hits per run for this group is 
5.22. 





For the group that was checked ten days after 
the date the sheets were filled out, there are 835 runs 
with a negative deviation of 82, and an average number 
of hits per run of 4.90. (These results are shown in 
Table 2.) 


Series [I-A included 850 runs, 425 of which were 
dhecked one day after, and 425 of which were checked ten 
days after this date. The series checked after one day 
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Results of Series II, showing the subjects used, number 
of runs of each subject, deviations from chance expecta- 
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TABLE 2 

















tion, and totals. 
ONE-DAY Check TEN-DAY Check 
Series Series 

Subjects Runs | Deviations Runs Deviations 
A.D. 50 + 20 50 - 35 
B.T. 25 - 18 25 ae 
D.C. 40 ¢ 40 - 4 
D.F. 50 ay 50 y= 
P.G. 25 dant | 25 - 12 
S.M. 25 - 10 25 - 1 
A.F. 50 + 34 50 - 7 
M.B. 50 + 18 50 + 8 
R.P. 25 + 12 25 oe 
J.G. 50 + 11 50 + 5 
B.S. 50 + 17 50 - § 
E.H. 50 + 10 50 - 18 
J.W. 50 o 50 + 11 
G.Z. 45 + 18 45 - 1 
E.S. 40 + 15 40  < 
I.M. 45 + 5 45 aig 
P.S. 25 ory 25 e.% 
B.C. 20 + 3 20 - 2 
H.W. 25 + 16 25 ~. a9 
H.R. 20 + 6 20 on 5 
A.A.F. 20 + 15 20 - il 
E.T. 25 - 7 25 - 13 
M.S. 25 + 8 25 + 10 
L.P. 5 e: 3 5 ce 
Total 835 180 835 - 82 

















has a positive deviation of 99, and the average number 
The series checked after ten 


of hits per run is 5.23. 
days has a negative deviation of 13, and the average 


number of hits per run is 4.97 (see Table 3). 


Series II-B included 820 runs, 410 of which were 
checked one day after and 410 of which were checked ten 
days after the date they were filled out. 
checked after one day has a positive deviation of 82, 

and the average number of hits per run is equal to 5.197. 
The series checked after ten days has a negative devia- 
tion of 66, and the average number of hits per run is 
4.83 (see Table 3). 
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TABLE 3 


Series II: Showing runs, deviations, average hits per 
run for A and B under the two conditions. 






































ONE-DAY Check TEN-DAY Check 
Runs | Deviation] Ave.Hits | Runs| Deviation | Ave.Hits 
; per Run per Run 
Series II-A| 425 } +99 4.23 425 - 13 4.97 
Series II-B} 410 +81 5.197 410 - 69 4.83 
Totals 835 +180 5.22 835 - 82 4.90 
C.R. = 3.11 C.R. = 1.42 


C.R. of difference (ONE-DAY and TEN-DAY) = 3.21 


The critical ratio for the ONE-DAY checked series 
of 835 runs is 3.11.. The P value for a chi-square treat- 
ment is .035. (See frequency distribution, Table 4.) 
The critical ratio for the TEN-DAY checked series of 835 
runs is -1.42. The P value for a chi-square treatment 


is .35. 


In Table 4 it will be seen that a score of 14 
occurs; the binomial expectation of this is .000,063. 
Also there are 13 scores of 11 or larger. The critical 
ratio of the deviation of these scores from the binomial 
hypothesis is 3.9. These items are of interest in sup- 
porting the deviation-critical ratio evaluation, even 
though they are obviously selected. 


For the sake of completeness, it may be of in- 
terest to pool the results of Series I and Series II. 
The total 3100 runs have a total positive deviation of 
122, which is insignificant (C.R. = 1.10). The differ- 
ence in experimental conditions between the two series 
and between the ONE-DAY and TEN-DAY sub-divisions war- 
rant the separate analysis given above. If a combination 
evaluation is desired, the chi-square method is the only 
one affording due distinction to the separate divisions, 
Using this method for the four subdivisions, ONE-DAY and 
TEN-DAY, both for Series I and II, a chi-square of 12.0 
is obtained, which with 4 degrees of freedom gives a 
probability of .017. 


A safe and approximate probability of the occur- 
rence of a block of results, such as that of the Series 
Il, ONE-DAY tests which gave a critical ratio of 3.11, 
may be found simply by multiplying the probability re- 
presented by 3.11, which is .00094, by 4 (since there 
are 4 such subdivisions in the report). The result, 


.00376, is significant; it represents the probability 
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TABLE 4 


Frequency distribution of run scores for Series II - ONE-DAY Check 
- by individual subjects and with the total distribution for the 
experimental series. 



































Sub- Runs and 
jects Frequency of Run Scores Dev. 
Of i] 2] 3] 4] 5] 6) 7 8) 9110/11) 12/13/14 

A.D. | Of 1] 3] 3] 14) 9] 6] 6] 2) 3] 1] 2] 0] 0] O] 50: + 2 
BT. | 0} 1) 3} 4) 7] 4] 4] OF} 2} OF Of OF] OF Of O| 2: - 18 
pc. | 0} 1) 2} 6) 7] 8] 5) 7] 2) 2] 0] 0] 0} 0} O}] 40: + 4 
DF. | 0} O} 4) 11) 6] 9] 6] 8] 5) 1] 0} 0] OF 0] OF} 50: + 1 
P.G. | 0} 2) 2} 3) 2] 5} 5] 3] 1] 1] 1) O] OF OF O} 25: + 1 
S.M. | 0} 1; 0} 8} 4] 5} 3] 2) 1) 1) OF 0] 0} OF O} 25: - 10 
ALP. | 0} 2} 2} 6] 6) 10} 3] 12) 4) 2) 0} 2] 1) 0) O| 50: + 34 
M.B. | 0} 0} 4) 4] 8] 12) 7] 8 5) 2} 0] 0] 0] 0] O| 50: + 18 
RP. | 0} 2) 1) 1) 4] 5] 4) 3) 3} 1] Of 1] 0) 0] O} 25: + 12 
J.G. | 0} O} 2} 5] 11] 11) 10] 6) 4) 1) 0) 0] 0} OF} O}| 50: + 11 
B.S. | 0} 1) 3} 8] 7] 6] 9} 9} 4) 1) 2] 0] 0} 0} O} 50: + 17 
EH. | 0} O} 5) 9] 11) 8) 5) 1) 4) 3} 1) 3] 0] 0] O| 50: + 10 
J.W.. | 0} O} 4) 8] 10) 11) 6] 5) 4] 2) 0] 0) 0] 0] O} 50: - 2 
GZ. | 0} 11 1) 3) 6] 14] 8] 9} 2) 0} 1) 0] 0] 0] 0} 45: + 18 
ES. | 0} 0} 6) 2) 5] 9] 7} 5) 2] 1) 3] 0] 0] 0} O| 40: + 15 
IM. | 0} 2} 2} 2) 11] 8{ 12) 3) 4) 1) 0] 0] 0] 0] O} 45: + 5 
P.S. | 0} 1) 2} 3) 4] 6) 2] 2) 4) 1] Of] 0] OF OF O] 2: + 2 
B.C. | 0} O} 1] 5} 2] 4) 5] 1) 1) 0] OF 0] 0] 1) O}] Ws + 8 
HW. | 0} O} 1] 5] 7} 4) 3] 2] 1) 3) 1] 0] 0} O] 1] 25: + 16 
H.R. | 0} 2} 0} 3} 8] 8} 8] 2 1] 2} 1) Of] 0] O] O} 20: + 6 
A.A.F| 0} 1] 1) 1] 5] 2) 4] O} 3} 0} 2} 1) 0) 0] O} 20: + 15 
E.T. | 0} 0} 3} 3] 6] 4) 7] 1) 0] Of] 1] 0) 0] 0} O|] 2: - 7 
M.S. | 0} 0} 2} 5) 5] 3] 3] 1] 3] 1] 1] 1] 0) 0] O| 25; + 8 
LE. | 0} 0} OF O} 1) 2) 2] 0} OF 0} 0} 0} OF 0) O %e¢ 1 
Total 0/18) 54/108 }152 |159|129 | 96/62/29)15}10; 1) 1) 1| 835: +180 


















































x2 = 17.60 with 9 d.f.; P = .09517 


that at least one of the 4 sections of results would have 
ate & positive critical ratio by chance as large as 
11. 


A COMPARISON OF SERIES I WITH SERIES II 


Subjects. Sixteen graduate women subjects took 
part in Series 1. Of this number, only four had ever 
been subjects in any ESP experiment before. For series 
II-A, eighteen subjects, all new, were used. Series 

II-B included twelve subjects from Series II1-A, five sub- 
jects from Series I, and one new subject. The entire 
group of subjects makes a total of thirty-five. They 
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form a fairly homogeneous group in most respects, since 
there are no wide deviations in age, occupations, and 
cultural standing. 


Material. The material, standard ESP record 
sheets and cards, were identical in the two studies, as 
was the recording and scoring of the data. Subjects 
were given the record sheet which they filled out and re- 
turned to the experimenter, who then took them to the 
record librarian. 


Order of Subjects' Calls. In Series I the sub- 
jects were explicitly asked to fill out the CALL column 
under number 1, then the CALL column under number 6, the 
CALL column under number 2, then number 7, etc. The ex- 
perimenter overlooked this point in Series II. 





Length of Series. In Series I, the subjects were 
asked to do ten runs of twenty-five trials for each of 
ten consecutive days. 





In Series II, the subjects were asked to do ten 
runs of twenty-five trials for each of five consecutive 
days. After an interval of three days, Series II-B was 
begun. The subjects were asked to do ten runs of twenty- 
five trials for each of five consecutive days. Thus, in 
Series II the ten runs were divided into two groups of 
five consecutive days each with a three-day interval be- 
tween, but in Series I the ten runs were done on ten con- 
secutive days. 


The Subject's Knowledge of his Scores. The sub- 
jects in Series II' were told daily of their scores and 
rewards, but a condition of Series I was that the sub- 
jects should not know their scores until the completion 
of the experiment. This meant that they were told noth- 
ing of the experiment except for their initial instruc- 
tions until the twentieth day after they began filling 
out the ESP record sheets. 





Rewards. The rewards were identical for both 
studies, except for the effect of nearness. In Series 
II the subjects knew their scores on the ONE-DAY tests 
within 24 hours, and accordingly the rewards were less 
remote than in the TEN-DAY runs and still less so than 
in Series lI. 


The Subject-Experimenter Relationship. The sub- 





ject-experimenter relationship was similar in both 
studies so far as the instructions and the tasks are 
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concerned, but there was more freedom and the contact 
was greater in Series II because the conditions permit- 
ted discussion of the daily scores, etc.* 


Table 5 is a summary of the comparison of Series 
I and Series II. 














TABLE 5 
A Summary of the Comparison of Series I and Series II 
ONE-DAY Check ONE-DAY Check TEN-DAY Check 
Subjects told Subjects not Subjects not 
daily scores told scores told scores until 
until end of end of experiment 
experiment 
Ave. Ave. Ave, 
Hits Hits Hits 
No. Per | | No. Per No. Per 
Runs| Dev.| Run Runsj Dev.| Run Runs| Dev. | Run 
Series I 715 |- 4} 4.99 715 | +28 | 5.04 
Series II | 835 | +180) 5.22 835 |-82 | 4.90 









































C.R. - Series I ONE-DAY Check = -0.07 ’ 
C.R. - Series I TEN—DAY Check = 2% C.R. diff. = 0.42 


C.R. - Series II ONE-DAY Check = Pi} 
C.R. - Series Il  TEN-DAY Check = -1.42} ©-R. diff. = 3.21 





DISCUSSION 


It was stated that three problems would be con- 
sidered in this study. They are: (1) Can pre-shuffle 
card-calling be demonstrated by normal subjects to an 
extent greater than chance expectation? (2) If so, how 
does success at calling the order of cards one day ahead 
compare with calling them ten days ahead? (3) If these 
two intervals of time affect the scores differently, how 
is this difference to be explained? These problems will 
be considered in the order listed. Finally, the sugges- 
tion of conditions probably favorable to the testing of 
the ESP hypothesis, and a consideration of further ex- 
perimentation as an outgrowth of this study will be dis- 
cussed. 





* Dr. Rhine reminds the experimenter that she had wanted to do the 
experiment as in Series II at the start but had been urged to do 
it as in Series I to insure equal motivation for the ONE-DAY and 
TEN-DAY tests. Her own interest was obviously greater for the 
Series II method. 
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The Demonstration of ESP in Pre-shuffle card- 
calling Tests. Any consideration of this problem in- 
volves an examination of the counter-hypotheses. For 
discussion, the counter hypotheses have been grouped 
into four broad categories: (1) the results are acciden- 
tal, (2) sensory perception was not wholly eliminated, 
(8) the results are due to errors in the scoring and re- 
cording, and (4) the results are due to the incompetence 
and untrustworthiness of the experimenter and others in- 
volved in the recording and scoring. The first of these 
is concerned entirely with the statistical evaluation of 
the results; the other three raise questions about the 
adequacy of the experimental conditions and the compe- 
tence of the experimenter. 








(1) When the commonly accepted measures of prob- 
ability are applied to the data designated as the ONE-DAY 
Check, under a chance hypothesis such results will occur 
but one time in a thousand. The repetition of this ex- 
periment with a critical ratio of 3.21 of the difference 
between the ONE-DAY Check and the TEN-DAY Check series 
by chance alone is likely to happen only 7 times in 
10,000. As already stated, there was no selection of 
the data. All the record sheets used in the two serigs 
are reported in this study with the exoeptions noted. 
All mathematical evaluations were independently checked. 





8. In order to make certain that there is no selection of data, 
every record sheet issued by the librarian to the experimenter 
for this experiment must be accounted for. In those cases in 
which the subject failed to fill out his sheet, the blank re- 
cords were returned to Mr. Gibson. Several of the record sheets 
were filled out by the subjects but were turned in too late to 
be stamped by Mr. Gibson. The experimenter has kept a record 
of all these sheets, even though it was stated that no sheets ex- 
cept those which were stamped on time by Mr. Gibson are con- 
sidered in the total. The following table shows by subjects, 
series, date, runs, and deviations the record sheets which were 
filled out, but not stamped. 


























Series} Date | Subject] ONE-DAY | Dev.| TEN-DAY | Dev. 
Check Check 
II-A | April 11 LE. |7 4445 -13 113341 a | 
II-A | April 12 LE. |7 4503 59772 
II-A | April 13 LE |48364 37335 
II-A | April 13 G.Z 155284} -1)64545 -1 
II-B | April 21 B.C. |511642/)] +3] 54487 +3 
II-B April 21 E.S. 3 6 3 2 2 = 8 ; 2 > «. 6 ee 
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fn order to obtain a series of scores to serve 
as a control on the chance factors inherent in the ex- 
periment, with the experimental factor (ESP) eliminated, 
a cross-check was made by checking the subjects' calls 
against card records of two days ahead--records they 
were not intended to match. This was done for the 835 
runs of the ONE-DAY part of Series II. An average score 
of 4.9952 was obtained, and a critical ratio of .07. 
The chi square of the score frequency distribution gives 
a probability of .63. The standard deviation of the 
cross-check distribution is 1.907, which is not signifi- 
cantly different from the theoretical value of 2.0. 


From this it may safely be inferred that inade- 
quate shuffling, chance coincidence of card sequences 
with subject preferences, or habit patterns, and any 
other factor associated with the randomness of the test 
material can safely be dismissed as explanatory hypothe- 
ses in this instance. This position is strengthened con- 
siderably by the many cross-check analyses of similar 
character made by earlier experimenters, all leading to 
similar results.’ Likewise, subjects could not have 
discovered any possible pattern trends in the cards, had 
there been any, and have followed them, since they did 
not see either the cards or the records at any time. 


There was no improper selection in the determina- 
tion of the stopping point of the experiments. The final 
subseries II-B had been begun with a definite assignment 
of record sheets, and when this was finished, there was 
no further time for continuance before writing and sub- 
mitting the report to the Graduate Office in time for 
acceptance. 


(2) The procedures of the experiment required 
that the subject fill out his individual record sheet be- 
fore the card orders against which the calls could be 
checked were determined. Therefore, it is clearly in- 
possible that the results of this experiment are explain- 
able by the hypothesis of sensory perception of the sub- 
jects. 


(3) The stamping of the CARDS in columns adja- 
cent to the CALLS by the librarian, and the independent 
checking and recording of the hits should be sufficient 
to eliminate the hypothesis of errors in scoring and re- 
cording as a possible explanation of the results. 








9. Greenwood, J.A., and Stuart, C.E., "Reply to Dr. Feller's Cri- 
tique," see page , this issue of the Journal of Parapsychology. 
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(4) It did not occur to the investigator at the 
time of planning and conducting the experiment as neceg- 
sary to assume that any of the persons participating in 
the recording and checking could be suspected of dis- 
honesty, inasmuch as the record sheets were at different 
times entirely within the keeping of each of the three 
persons concerned. Therefore, it is obvious that out- 
right dishonest practices with regard to the data could 
have been possible. But certain features of the pro- 
cedure make it highly unlikely that unconscious errors 
in recording or the loss of data of a "motivational" 
character could have occurred. The first such feature 
was the use of numbered record sheets. All but eight 
of such sheets used by the experimenter in any experi- 
ment by her have been returned and acknowledged by the 
record librarian. Those eight sheets have been accounted 
for by the subjects to whom they were issued in state- 
ments filed with the librarian. A second feature was 
the procedure of using a hectograph stamping of all the 
sheets in this experiment by the librarian. A third 
feature was his use of a shuffling machine rotated a 
fixed number of times to determine the order of the 
cards. A fourth feature was the separate recording of 
the run scores and deviations by Miss Maxine Slaughter 
and the experimenter. A fifth feature was the independ- 
ent computation made by Miss Esther Bond and the experi- 
menter. 


There will be, no doubt, other objections than 
the four main hypotheses usually brought against extra- 
sensory perception to the acceptance of the precognition 
hypothesis. For instance, it may be suggested that the 
mechanical shuffler does not wholly eliminate the ESP 
shuffle. However, since the machine was rotated a fixed 
number of times for each deck (3 times), this counter- 
hypothesis in its usual form is not applicable. So far 
as can be judged, if extra-sensory perception did take 
place, it must of necessity be explainable by hypotheses 
other than those of clairvoyance and telepathy, which 
were eliminated by the conditions. The evidence suggests 
the validity of the hypothesis of precognition, i.e. that 
significant pre-shuffle card-calling of extra-sensory 
and non-inferential character has occurred, the calling 
corresponding to random future events; but no final con- 
clusions can be reached without much more evidence. 


If, however, we accept the results at their face 
value for the moment and assume the hypothesis of pre- 
cognition to be demonstrated, what inferences as to the 
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nature of this function are we able to make? Has any- 
thing been learned regarding the nature of so-called pre- 
cognition? In view of the fact that the writer considers 
that the results are of a suggestive rather than a con- 
clusive nature, conclusions to be reached regarding se- 
condary problems must be even more tentative. One find- 
ing, at least, relates pre-shuffle card-calling to other 
modes of ESP: methods successfully used in the demonstra- 
tion of telepathy and clairvoyance may be used to measure 
pre-shuffle card-calling. The DT, or calling "down 
through" a pack of cards method was utilized in this ex- 
periment with only the addition of the condition that 

the key decks against which the calls were checked were 
not to be arranged nor recorded until some time after 

the calls were made. 


How does the use of different time limits in the 
same experimental situation affect pr¢jshuffle card- 
calling? It has been stated by Rhine 0 that precognition 
is a logical inference arising from distance experiments 
in ESP. That is, when distance has been a condition in 
ESP experiments, there has been no critical effect of 
space shown in the results. Since time measurements are 
made in reference to spatial relations, and our concep- 
tions of time and space are inseparable, the probability 
would seem to be that distance in time would likewise 
not affect ESP scores one way or the other. 


At first glance, the results obtained in Series 
II seem to show that pre-shuffle card-calling by subjects 
to an extent greater than chance expectation can be demon- 
strated as taking place within a twenty-four hour limita- 
tion, but will not extend as far in time as ten days. 
If this were true, it would indeed be a fact of great 
importance about the nature of precognition; a fact that 
effectively contradicts our logical inference regarding 
space and time. However, a closer examination of the re- 
sults suggests that such an interpretation may be incor- 
rect. Possibly the explanation of the difference might 
be found in the consideration of the subjects and their 
reactions to the different testing conditions for the 
demonstration of ESP ability. An expansion of this argu- 
ment follows in the next two divisions. 


The Contrast of Difference of Time Intervals in 
Checking. The results of Series I gave no significant 
ifference for the ONE-DAY and TEN-DAY checks. The 








10. Rhine, J.B., "The Effect of Distance in ESP Tests," Journal of 
Parapsychology, September, 1937, Vol. 1, p. 184. 
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ONE-DAY checked series of 715 runs gave a deviation of 
-4; the TEN-DAY checked series of 715 runs a deviation 
of +28. In fact, the results of the series as a whole 
were clearly explainable by chance, comparable to the 
TEN-DAY check half of Series II. 


A striking difference is indicated in the results 
of Series Il. The series of 835 runs designated as ONE- 
DAY check has a critical ratio of 3.11, and the average 
number of hits per run is 5.22, while the TEN-DAY check 
series of an equal number of runs gave a negative criti- 
cal ratio of 1.4 and the average number of hits per run 
is 4.90. This difference between the ONE-DAY and TEN- 
DAY checking intervals yields a critical ratio of +3.21. 
(See Table 5.) 


The Interpretation of the Results Obtained in the 
Contrasted Time Intervals of Series II. A suggestion was 
made that the difference between the ONE-DAY and TEN-DAY 
groups of Series II might be due to the fact that the 
five calls for the ONE-DAY group were made first in most 
instances while those of the TEN-DAY group followed. A 
difference in procedure between Series I and Seriee II 
makes it possible to use the results of Series I as a 
control for the effect of the order of making the calls 
in Series II. 








In order to test whether the sequence of calls 
did affect the results, an investigation of the results 
according to the order of the subject's calls for Series 
I was made. 


As has been stated, in Series I the subjects made 
alternate calls for the ONE-DAY and TEN-DAY conditions. 
A comparison of the first five runs made by the subject 
and his second five runs was made. The comparison was 
made by totaling all the runs of all subjects under col- 
umns 1-10, and comparing the sum of 1, 2, 3, 6, 7, se- 
lected because they were the first half of the runs made, 
with the sum of 4, 5, 8, 9, 10. The total hits of the 
first five runs (columns 1, 2, 3, 6, 7) filled out by 
the subject and disregarding the checking time division 
equals 3,512. For the second five runs the total of the 
hits equals 3,662. The difference of 150 (S.D. of the 
difference is 75.6; C.R. is 1.97) is not significant. 

It is concluded that there is no appreciable difference 
shown between a subject's first five runs and his second 
five, and that in Series II the difference must be due 
to some other condition. 
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Any explanation of the contrasted results of the 
time intervals in Series II may involve as well an ex- 
planation of the different results obtained in Series I 
and in Series II. As was outlined in the treatment of 
the results, the whole of Series I can be classed as de- 
layed for the subjects in the same sense as the TEN-DAY 
series of Series II. That is, the subject's knowledge 
of his scores was delayed in both instances. It is sug- 
gested that he had no clear differentiation of the time 
intervals in Series I, since he knew nothing of his 
scores until all the data were in. This meant his know- 
ledge of all his scores was delayed for a period of more 
than twenty days after Series I began. His knowledge of 
his daily scores in the TEN-DAY series was delayed for a 
period of ten or more days after Series II began. But 
in Series II, this lack of knowledge of his daily scores 
on the TEN-DAY series is in clear contrast with his 
daily knowledge of scores and rewards for the ONE-DAY 
check group. The results seem to suggest that the know- 
ledge or lack of knowledge made for a differentiation in 
his mind of the time intervals in Series JI -- a differ- 
entiation which was lacking in Series I -- and that the 
explanation of the difference in results may be traced 
to this factor. 


A "negativistic" attitude of the subject towards 
tests in which delayed checking is a condition has been 
previously noted. Rhine says, in the summary of his 
article "ESP Tests with Enclosed Cards," "It was found 
that there was a marked correlation between delay in the 
checking and decline in the scoring." This decline 
effect appeared in six of the seven test procedures used 
in the experiment. 


The subject, his interests and motives, must be 
counted as variables in any psychological experimenta- 
tion. It is believed that in the most successful ESP 
experiments, the subject should know exactly what he is 
doing and what his results are at the end of one run or 
one unit of runs. This is perhaps of particular impor- 
tance in ESP testing because the subject cannot be said 
to possess much cognitive clarity about the very nature 
of his task. He is asked to call symbols spontaneously 
so that they will match a pattern in someone's mind, or 
on a slip of paper, or as they will be recorded at some 
future time on a slip of paper. His introspection while 
he works or afterwards tells very little about his 








11. Rhine, J.B., "ESP Tests with Enclosed Cards," Journal of Para- 
psychology, September, 1938, Vol. II, p. 199. 
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conscious direction of his task, if indeed there can be 
said to be a conscious process important in ESP. It is 
certain that the subject feels the situation to be more 
real, and as possessing more ego-reference when he is 
daily told of his scores and encouraged to do well. 


The results of Series II clearly support this 
line of reasoning. The runs which were immediately 
checked with the subject's knowledge of scores gave\a 
positive deviation (835 runs + 180), an average number 
of hits per run of 5.22. Those which were checked after 
ten days and which possessed no immediate interest for 
the subject gave a negative deviation (835 runs -82), 
an average number of hits per run of 4.90. Experiments 
with sealed cards in which the subject's knowledge of 
his segpes was delayed are cited in confirmation of this 
point. 


A Consideration of Other Factors Favorable to 
the Testing of the ESP Hypothesis. Other factors than 
the one of the subject's knowledge of his daily scores 
may have contributed to the difference in the results of 
Series I and II and are of importance in a consideration 
of further testing of the precognition hypothesis. 








The experimenter believes that the effect of the 
subject's knowledge or lack of knowledge regarding his 
daily scores is the probable explanation, but desires to 
state certain other differences in the conditions of 
Series I and Series II which may have been factors: 
namely, (1) the length of the series in consecutive test- 
ing days; and (2) the subject-experimenter relationship. 


(1) In Series I, the subjects were told the ex- 
periment would run ten days, and were given ten record 
sheets to fill out. Each record sheet represented 250 
trials. Those subjects who did 100 runs -- who filled 
out a record sheet for each of the ten days -- made by 
the end of the experiment 2500 trials each. Since the 
experimental days were consecutive, it may be the series 
was too long for the best results to obtain, although 
there was no marked difference between the scores for the 
first day and those for the last in the series. 


In Series II-A, the experimenter decided on the 
grounds of personal convenience and close supervision 0 
the experiment that it would be preferable to reduce the 
consecutive den-day period to a five-day block. The 





12. Ibid., p. 199. 
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subjects were asked to do ten runs of 25 trials each on 
record sheets for five consecutive days. The block be- 
gan April 9 and continued through April 13. The check- 
ing of the TEN-DAY group, of course, did not begin un- 
til the tenth day from April 9, April 19. A second 
block, (II-B) was begun after an interval of three days. 
It ran from April 17 through April 21, with the checking 
of the TEN-DAY group beginning ten days from April 17 ; 
(April 27) and continuing through May 1. 


(2) The experimenter feels that there was a 
marked difference in the kind of contact she had with 
the subjects during Series I and Series II. In all psy- 
chological research, the personal element assumes im- 
portance, and in an experimental task which involves two 
people, the relationship between the two is a factor 
which must be considered. It might be argued that the 
conditions of the pre-shuffle card-calling experiment 
eliminated that nebulous sort of relation we sometimes 
call "rapport," and that the reward scale was the im- 
portant motivating force. The experimenter believes 
that the establishment of a certain mutual interest in 
the task and a feeling of unification is as important to 
the securing of high scores in pre-shuffle card-calling 
as in other branches of ESP research. It is her belief, 
as well, that a reward scale is merely an added incen- 
tive, and perhaps serves as a means of synthesizing and 
openly expressing the several motives in the situation. 


The experimenter had little contact with most 
of the subjects in Series I beyond the initial instruc- 
tions. In Series II the experimenter had daily contact 
with all but two or three of the subjects. It is her 
belief that the attitudes of the subjects toward the 
wo “ more favorable than those of the subjects in 

eries I, 


Consideration for Further Experiments. The sug- 
gestion that the subject's interest and motivation aris- 
ing from his knowledge of his scores and several other 
favorable factors are responsible for the difference in 
the resuits of the contrasted time intervals and make 
for more favorable testing conditions should be further 
elaborated experimentally. 





Particular factors to be considered in the ex- 
periment to be outlined are: (1) the subject's cogni- 
tive clarity concerning the experiment, (2) the subject's 
interest and motivation, (3) the subject-experimenter 
relationship, and (4) the length of the series. 
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TENTATIVE CONCLUSIONS 


The following tentative conclusions are made: 


(1) That in the ONE-DAY check series of Series 
II there seemed to be a positive demonstration of extra- 
sensory perception, and that the hypothesis of pre-cog- 
nition used to describe this demonstration warrants more 
research. 


(2) That the difference in the results of the 
contrasted time intervals, in Series II, may be explained 
by the difference in interest and motivation of the sub- 
jects towards the two conditions. 


(3) That the difference in the results of Series 
I and Series II may be due to the poorer differentiation 
in the mind of the subject of the one-day and the ten-day 
time intervals in the former, and the clearer contrast 
in the latter series. 


(4) That the effect of telling the subject his 
daily scores probably has a direct relation to (a) the 
interest and motivation of the subject under the differ- 
ent conditions, and hence to (b) the difference obtained 
in the results of the contrasted time intervals. 


Reno, Nevada. 
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STATISTICAL ASPECTS OF ESP4 


Willy K. Feller 
Brown University 


1. Presumably many a student feels puzzled by 
the frequent controversies arising out of statistical 
investigations in various fields. In fact, statistics 
makes claim to mathematical rigor, and still its practi- 
cal agplications are often disputed or rejected as ab- 
surd.~ I fear it may sometimes produce a feeling that 
mathematics is, after all, ‘une de ces voies rationelles 
et scientifiques, qui n'est qu'une etroite, courte et 
sale impasse, ay fond de laquelle on se casse le nez in- 
glorieusement'. The fact is that the importance and 











1. Invited address, delivered at the Imke Mathematical Seminar, on 
April 24, 1940. The publication of the address was not original- 
ly intended, but was suggested both by the Mathematics Depart- 
nent of Duke University and by Professor Rhine. Professor Rhine 
and other interested persons have insisted that it-would be de- 
sirable to publish the address in its original form, and I have 
done ny best to conform to this wish: for this, however, I had 
to rely on incomplete stenographic notes taken during the lec- 
ture. The following are the only essential changes: 


(i) The original lecture had been planned as a positive 
contribution and, accordingly, all was avoided that could be in- 
terpreted as criticism of individuals; in particular, the lec- 
ture did not contain any reference to or quotation from the cur- 
rent literature, except a tribute to C. Leuba. Subsequent events 
have induced me to change my attitude, and I have hesitantly, sub- 
stantiated my considerations by a few examples. 


(ii) The content of s2 was discussed only in private conver- 
sations in the Parapsychology Laboratory. 


(iii) Time did not permit completing the last section: the 
two mathematical examples and the very end were omitted.— Two 
further incidental departures are marked as such. The new book 
by Rhine and his collaborators (1) appeared after the present 
lecture. 

An outstanding biologist assured me that, in periods of depres- 

sion, he is accustomed to recover by reading some modern statis- 

tical investigations in biology or medicine. They give him, he 

explained, the comfortable feeling of certainty that Nature 

would never allow anything to happen that would be probable to 

a statistician. 

3. "Les opinions de M. Jerome Coignard" by A. France. (Oeuvres com- 
pletes, VIII, Paris, Calmann—-Levy ed., 1926, p. 376.) 
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indisputable successes of mathematical statistics have 
induced many writers to describe the statistical methods 
empirically without due attention to the underlying the- 
ory. Thus it happens that frequently formulas are given 
and used without discussion of the fundamental assump- 
tions and far beyond the field of their original scope 
and their applicability. 


The statistical methods involved in ESP-researgh 
in particular have been the subject of ardent discussions 
It must be admitted that many criticisms were biased or 
concerned-with minor details. On the other hand, ESP- 
advocates have rejected well-founded and almost obvious 
criticisms. Today, as far as I can see, nobody denies 
any longer that the fundamental law of probability, on 
which all numerical evaluations rest, has been, until 
quite recently, consistently misinterpreted, and that 
most of the experiments in ESP were not conducted in ac- 
cordance with the theoretical requirements. I1 am refer- 
ring to the so-called effect of optional stopping, which 
has been pointed out by several critics. It is a most 
surprising chapter in the history of science that an ar- 
dent wishful thinking could have misled the ESP advocates 
to deny vigorously an effect which is quite obvious, well 
known in ekperiganse and the subject of a vast literature 
in probability. 


We shali see in ss6-7 that this discovery funda- 
mentally changes the aspect of the empirical material 
available at present; however, before proceeding to dis- 
cuss this main point I propose to analyze some more tech- 
nical details, especially the problem of shuffling 
(ssE-5). 





4. E.g., the tests for goodness of fit, designed to estimate cer 
tain parameters of probability distributions, are used frequent- 
ly in cases where there is absolutely no place for probabili- 
ties; we are, in such cases, advised about the numerical values 
of the odds in favor of such and such assumption, though these 
words have no sense whatever. For an actual case where such 
statistical theories confirm with high odds three contradictory 
assumptions cf. Feller (5). 

5. For details and references see below, s6. As far as I am aware, 
among all experiments supposed to show the existence of ESP the 
first and only cone to take into account the effect of optional 
stopping is that by Pratt and Woodruff (16). According to these 
authors (loc. cit., p. 141) "the optional stopping correction 
was discovered as the present research was nearing completion." 

6. The reading of ss3-5 is not necessary for the understanding of 
the sequel. 
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Now I wish to make it perfectly clear from the 
very beginning that the whole interest of the following 
discussion is concentrated on the statistical methods 
involved in ESP research. In particular, I am not going 
to discuss the experimental side, and for the purposes 
of this discussion it may be assumed that the _experiments 
have been performed in the best possible way. Nor is 
the aim of this discussion to "explain" anything or to 
prove that ESP does not exist: This obviously could not 
be achieved by purely statistical arguments. It may be 
clearly stated that a few experimental series are re- 
corded with so large deviations from chance that they 
obviously never could be reasonably "explained" purely 
statistically. However, results of this type form but 
an exceedingly small portion of the statistical material. 
On the whole, the most surprising feature of the material 
is its extraordinary heterogeneity... Subjects with sup- 
posed ESP abilities lose and regain their abilities very 
rapidly. We hear also of series of guessings consistent- 
ly below chance. Furthermore, there is a great difference 
between individual experimenters, which has been the sub- 
ject of several discussions and investigations: some 
experimenters are particularly lucky and find some degree 
of ESP ability on the average in every fifth person test- 
ed; in many of the successful experiments subjects 
guessed at a distance, in some of them even temporally 
in advance of the actual card-shuffling (implying a kind 
of precognition’). Other investigators, on the contrary, 
were unable to find any traces of ESP ability: 





This in itself certainly would not affect the 
ESP hypothesis. However, we shall see that the fluctua- 
tions of the score-level caused by chance are bound 





7. It is of course a pure question of terminology whether one says 
"the experiments have not been performed according to the re- 
quirements of the theory", or "the theory has been misinterpret- 
ed." With the first formulation Professor Camp's well-known 
statement (J. Parapsychol., 1937, I, 305) holds also after the 
discovery of the so-called optional stopping effect. 

8. That is to say, no reasonable allowance for the effects discus- 
sed in the sequel would essentially change the aspect of these 
series. The two record series obtained by Riess (22) (1850 
trials with 1349 hits against an expected value of 370) and by 
Martin and Stribic (14) (50,000 trials with 14,280 hits against 
expected 10,000) are not even subjected to the effect of option- 
al stopping: in fact, the Riess series was terminated by an ill- 
ness of the subject with consequent loss of her guessing ability, 
and in the Martin-Stribic series the number of trials was, ac- 
cording to the experimenters, fixed in advance. 

9. Cf. Rhine (20) and Rhine-Smith-Woodruff (21). 
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sometimes far to exceed the usually accepted limits, and 
that a convenient use of the method of optional stopping 
completely changes the situation: to my mind there is 
not the slightest doubt that this accounts for a part of 
the divergence in the experiences of different investi- 
gators. e shall indeed see that a considerable portion 
of the so-called significant material can and must be 
ascribed to chance, and that sometimes conclusions on the 
nature of ESP were drawn from material including no single 
significant case. It seems to me that the statistical 
material available at present affords but one safe con- 
clusion: that ESP ability is, at its best, extremely 
rare. 


To summarize: it is my definite impression that 
at present the evidence in favor of the ESP hypothesis 
consists of the few series with extraordinarily high 
scores: that any defense of and any attack on the hypo- 
thesis should be based on this portion of the material 
alone. I do not think that the ultimate word belongs to 
statistics: in this respect experiences of the past 
strongly suggest extreme caution. This extremely cau- 
tious attitude of most statisticians is also stated by 
the English statistician Bartlett: "In card experiments 
with supposed telepathic or other paranormal subjects, 
most of us are probably still a little chary about attri- 
buting statistically gignificant deviations to extra- 
sensory perception, "1 And the point is made perfectly 
clear in the following passage from a letter of R. A. 
Fisher to H. Rogosin: "Perhaps I may say, with respect 
to the use of statements of very long odds, that I have 
before now criticised their cogency on the grounds, not 
only that the procedure of calculation is often question- 
able, but that they°are much less relevant to the estab- 
lishment of the facts of nature than would be a demonst ry 
tion of the reliable reproducibility of the phenomena." 


2. The first point which I wish to emphasize is 
the following: When summing up the evidence in favor of 
ESP, Rhine combines all published series of which he 
knows and which had been conducted under a specified set 
of conditions to exclude sensory cues into a series of 
907030 trials with 194605 hits;+2 and then treats this 


10. In a lecture before the Royal Statistical Society, cf. (3), 
p.» 14. The remark was not challenged in the subsequent discus- 
sion. (The publication came to this country after my address 
was delivered. ) 
11. vy] a was read by H. Rogosin at the ESP Symposium, cf. 
» De . 
12. For the convenience of the reader the figures are here given 
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series as a continuous experimental series with a criti- 
cal ratio (C.R.) of 39.90. It must be said that this 
procedure is unjustifiable, and that no mathematical or 
statistical reality whatsoever can be attached to this 
C.R. 


The same method is the common practice when an 
experiment is conducted with several subjects: it is 
argued that, if no observations are dropped, the trials 
form a continuous chance series unless ESP intervened. 

We shall see that, owing to optional stopping, not even 
this restricted statement is true. The case is, however, 
incomparably worse when a heterogeneous material scattered 
throughout the literature is combined into one statisti- 
cal series: this procedure could be justified were a 
so-called randog sample of the set of all experiments 

ever conducted.+3 Now the whole material ever published 
(with a selection only in view of reliability) is cer- 
tainly any thing but a random sample: would any one main- 
tain that it contains all negative experiments ever per- 
formed as it actually contains all positive experiments? 


Rhine argues/4 that he has avoided "selection of 
results" by taking "all the tests of which I had any 
knowledge". But the strongest possible selection lies 
in the very fact that the results have been published. 

It is like an army which is selected from only male indi- 
viduals and is therefore no suitable basis for statistics 
of sex-distribution. Indeed, every positive result is, 
rightly, deemed interesting and published with details; 
whereas the chance results do not “qgutve detailed pub- 
lication, and are seldom published. Nevertheless every 





(Footnote Continued) according to Rhine's new book (1), table 
7, pp» 95-97. In April this book was not published and my rea- 
soning was based on the older figures given by Rhine at the ESP 
Symposium, cf. (19), p. 255. The same method is used in dif- 
ferent places. 

18. Experience shows that even then the procedure would be extreme- 
ly dangerous and that getting a true random sample is much more 
problematic than it would seem. 

14, Loc. cit. in footnote 12. 

15. Rhine's table 7, mentioned in footnote 12 contains 34 entries; 
among these there are but four with negative deviations and six 
with positive deviations and a C.R. not exceeding 3. However, 
these entries do not contradict the argument that the material 
is selected. Indeed, one of the negative C.R. is purely formal 
since it "was pursued with that as goal" (Rhine, loc. cit., p. 
102: the C.R. is there properly treated as positive). All the 
six experiments with positive but insignificant deviations were 
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one of us has knowledge of experiments in card guessing, 
which have been performed at American universities and 
colleges, and have led only to chance results. Any in 
the very publications from which Rhine's figures are 
taken, we find recorded that also other experiments, under 
changed conditions, were undertaken but without positive 
results. 


Now it should be constantly borne in mind how 
easy it is to obtain series with an arbitrarily large 
C.R. by combining perfectly trivial random series. After 
all, it is easy to obtain any number of series consisting 
of 1000 trials each with a deviation of 12 hits above 
chance expectation: the C.R. would be 0.949 and the 
probability of such a series is about 0.18. But combin- 
ing 10 such random series into one we get a "significant" 
C.R. of 3, by combining 100 such series we arrive at a 
C.R. of 9.487 which indicates a "probability" of about 
10-29! Or let us toss a coin 25 times: the probability 
of obtaining at least 13 "heads" is obviously g. Suppose 
now that this experiment is repeated and that, for any 
reason, only runs of 25 trials with at least 13 heads are 
counted (=published). Every run represents a deviation 
of at least $ above chance expectation (125 heads) and 
the probable deviation exceeds 2. In order to get a 
series of the length of Rhine's table mentioned above2 
let us combine 36280 such runs: the probable deviation 
above the "chance expectation" (= 36280 =x 12.5) exceeds 
72560 and we are absolutely certain to obtain a deviation 
of at least 18140: thus the probable "C.R." would be 152 
and at the very worst we would get a C.R. of 38 (against 
Rhine's C.R. of 39.90). Yet we obtained our series by 
combining runs of a probability s and this without selec- 
tion after we have agreed to count (publish) only a cer- 
tain class of runs. And we should have obtained a simi- 
lar result by concentrating our interest, instead of on 
single runs, on longer experimental series with a favor- 
able result. 





(Footnote continued) conducted for special purposes: to com 
pare different methods or stimuli, examine the precognition 
hypothesis, covariation, etc. All six are considered as favor- 
able for ESP. ‘Two of the remaining three entries with negative 
deviations (Nos. 20 and 32) are not accessible to me. The last 
(No. 8) is one of eight series produced by Carpenter and Phalen 
(4). Seven of them show, in part considerable, significant 
positive deviations, and the experiment as a whole was published 
to confirm the ESP hypothesis (the seven favorable series are 
dropped by Rhine probably because of a possibility of sensory 
cues). — Thus the table contains at most (if any) two experi- 
ments with chance results, conducted in order to examine the re- 
peatibility of the Duke experiments. 
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Or to use a simpler language: Suppose we make a 
statistical study of the proportions of different parts 
of the human body by collecting all reliable statements 
and measurements, without selection, which are published 
in the clinical periodicals: our conclusions would ne- 
cessarily be queer. And the fault would not be ours: 
rather it would be that the writers in clinical period- 
icals concentrate their interest on exceptional cases. 


8, One of the most important assumptions under- 
lying the whole mathematical technique of ESP research 
is that all possible arrangements of the 25 cards in 
the deck are equally probable. Plausible as it sounds, 
this assumption is very abstract: for there are 
623, 360,743,125,120 different arrangements of the five 
symbols and to verify that they are, on the average, 
equally frequent, at least 10 experiments would be 
needed: if the whole of mankind including newborn babies 
devoted eight hours daily to card experiments, working 
at the rate of an eygetsnent a minute, it would take 300 
years to perform 10 experiments. But even if the ex-. 
periment confirmed the hypothesis, it would only follow 
that the usual estimates of the probable fluctuations 
dug, to chance hold in series with a length comparable to 
10*‘. To ensure that the estimates hold in practice, 
and in every case that might occur, much more is needed: 
it would be necessary to prove tnat the usual method of 
shuffling, when repeated, necessarily produces so-called 
independent random samples of the set of all arrange- 
ments. This means, e.g., that there is no linkage what- 
soever between the distribution of cards in a deck be- 
fore and after shuffling. And for the purposes of ESP 
it would not even suffice that the possible linkage is 
statistically negligible on the average over a large 
number of experiments (perhaps, to a mutual compensation 
of ditferent effects): it is necessary to know that the 
usual formula for the standard deviation, etc., holds for 
every subseries. 


Now, it is a well-known experience that ordinary 
card shuffling is far from producing a random distribu- 
tion. It is well-known that some card-players take ad- 
vantage of their know} edge of the actual card-dis tribution 
in the previous game; and it is also known that the 
most unexpected card-distributions are recorded to have 





16. Of course, this is possible only for very special sorts of in- 
terlinkage, and hence not for all shuffling methods. To ensure 
@ perfect game much less is needed than perfect randomness. 
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actually occurred.’ This may not sound convincing. 

But we have also the experience of the repeated attempts 
to produce empirically random series. The simple meth- 

ods of dice-throwing, qartyghs S*i8e- etc.,proved a fail]. 
ure and were given up.*°’ 


Now the reader may wonder whether these doubts 
are not expressions of a too sophisticated mathematical 
mind: whether any adverse effect could actually be 
traced in the ESP experiments. This question could, no 
doubt, be answered exhaustively were it not for the con- 
mon practice of not publishing complete records of the 
ESP experiments. As it is, I was able to trace in the 
literature only one complete record of actual card 


17. After all, doubts about the effectiveness of card shuffling 
have led to Poincare's investigations which, finally, gave the 
result that infinitely many shufflings would produce a random 
order. 

18. Compare the actual construction of the tables (29) and (11). 
The latter were constructed with the utmost refinement imagin- 
able. They contain 100 groups of 1000 random numbers each. They 
were subjected to four different tests of randomness. And it 
may surprise optimists that "although the complete entity of 
100,000 random digits is seen to satisfy these tests, five of 
the 100 groups are found to deviate from expectation when tested 
as individual groups". It should be noted that precisely this 
so-called "poor local randomness" would be crucial for the ESP 
experiments. 

19. Also the following personal experience may be recorded. The Ad 
ministration of the Telephone Works of Stockholm ordered, for 
purposes of random experiments, 100 numbered balls about 4 inch 
in diameter. The balls were of finest Swedish steel, the num 
bers almost invisible, and whatever care skilled craftmanship 
could take to ensure homogeneity was taken. Nevertheless we did 
not succeed in obtaining random series. 

Recently a friend of mine, an experienced card player, chal- 
lenged these considerations on the grounds of his personal experi- 
ence. So we decided to make experiments. Apparently he used 
one of the less effective methods, but I am told that the shuf- 
fling machines perform essentially the same method of shuffling, 
though probably with them the number of shufflings is larger. 
Now, he would shuffle the deck 7 to 12 times; the deck was then 
not cut but simply compared with the original distribution, or 
its reverse order according to whether the number of shufflings 
was even or odd. The number of coincidences varied between 10 
and 20 instead of the expected number of 5 (we used ESP cards), 
Even when the deck was cut I was able to guess consistently above 
chance, although only when I observed from afar the cutting pro- 
cedure in order to estimate at what place in the deck it was 
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distributions, that published by Riess (23). Moreover, 
Kennedy's paper (12) contains (opposite p. 216) a photo- 
static copy of a part of the original record in an ex- 
periment of Mac Farland; however, the card distributions 
only of 10 consecutive runs are given, so that it is only 
possible to analyze the effect of at most 9 shufflings. 


Now the card distribution in the Riess-series 
seems to be satisfactory, as far as I could make out with- 
out a thorough analysis. However, in this experiment it 
was a rule to perform only two runs on the same day, and 
the 37 days of experimentation are scattered over a peri- 
od of several months. It is therefore impossible to as- 
certain to what degree the randomness is due to excellent 
shuffling. Suppose, therefore, that in this case the 
shuffling was perfect: after an analysis of Mac Farland's 
10 runs no doubt will be possible that such is not always 
the case. 


Indeed, let us consider the last three runs. If 
the five symbols are denoted by c (circle), p (cross), 
s (star), r (square), w (waves) the actual card distribu- 
tions in the runs number 8, 9, and 10 were 


(83) rswpe srere pprcc wpwsw ssprw 
(9) wrsws wpess specrr cpprr wececwp 
(100) prrwe cwpes sswpp rerwr swscp 


It will be noticed that the cards show a marvelous ten- 
dency to cling together. In the 9th and 10th arrangement 
of the deck the 1, 2, ... places are occupied by the same 
symbols as the following places in the 8th or 9th arrange- 
ment of the deck, respectively, 


(8-9): 18119 202 345621 2223879 10 11 12 13 24 
25 14 15 16 17 


(910): 18 19 20 21 22 23242589 10116712 14131512 
345 16 17 


It should be borne in mind that the deck was cut: thus 
after the shuffling proper the first card of the rear- 
4 deck lay below the last one. In other words, the 
shuffling of the 9th arrangement of the deck left the 
group of cards numbered 16-25 (i.e., 10 cards) in the 
same order; in addition, the 5 cards numbered 1-6 clung 
together, likewise the 4 cards 8-11. In all 17 cards 


(Footnote continued) made. This is, of course, not the usual 
set-up in ESP experiments but proves nevertheless that shuffling 
does not necessarily produce randomness. 
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have after the shuffling the same successor as in the 
original deck. A similar remark holds for the shuffling 
in the 8th run: here 15 cards conserved their successor, 
and it will be remarked that even the same groups of 
cards show a tendency to cling together. The linkage be- 
tween consecutive decks is so strong that it can be ob- 
served even when the distribution of the 10th run is 
compared with that in the 8th run. We get the following 
reordering: 


(8+10) 12 138 242514 15161756 21223423 789 181 
19 20 2 10 11, 


and it is seen that despite a repetition of the procedure 
of shuffling and cutting the deck, 13 cards have conserve 
their successor. 


Strictly speaking, it cannot be logically proved 
that the cards actually did stick together: in a deck of 
5 different symbols and five cards of each there are 
(51)5 = 24,883,200,000 different arrangements of the canis 
showing the same arrangements of symbols, and only the 
latter is known to us. It is, however, apparent that 
chance alone produces rearrangements of a different kind, 
and the corresponding odds are easily computed. There 
is only one reasonable ambiguity: the cards numbered 24 
and 25 in the 9th run were actually the same as the cards 
6 and 7: it is therefore perfectly possible that the 
order of the cards was not, as described above, but 


(9-10): ooe 22 236789 iO 11 24 25 12... 


This, however, is of no importance, the number of cards 
with conserved successor being the same. (The largest 
unchanged group would now consist of 8 cards, the next 
two of 6 and 5 against 10, 5 and 4 with the first alter- 
native.) 


Not all runs are linked to the preceding one in 
the same net fashion, e.g., it is probable that the deck 
was changed, or fell to the floor, between the fifth and 
the sixth run. Between the 6th and the 7th run the group 
9-16 was conserved in its original order but for the 
intercnange of the cards numbered 14 and 15. The rear- 
rangement of the third run shows two such interchanges: 


($4): 21 22 23248 910112513 14 12 15 16 17 
18 191202 34567 
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where 14 cards conserved their successor, etc. After 
this example it will hardly be degied that linkages of 
the worst kind actually do occur. Nevertheless, I 
proceed to further examples. 


It has already been pointed out that I was not 
able to find records of any other card distributions 
which have actually occurred. However, there are two 
records from which at least indirect conclusions about 
the actual card distributign are possible. I was unable 
to find more such records. 


4. L. Warner (31), p. 237, published a complete 
record of 250 consecutive trials. It is unfortunate for 
the purposes of the present investigation that the cards 
used were not arranged in decks: from trial to trial 
"a different deck, newly shuffled, was cut to obtain the 
next card to be guessed." Obviously this procedure is 
much more favorable to randomness than the usual shuf- 
fling of a deck. The result is, however, not encouraging. 
Indeed, the order of symbols is expected to be random, 
each symbol having, in each trial, the probability 1/5. 
The actual distribution shows deviations from expecta- 
tion in different aspects. Suffice it to note that the 
symbol c occurs 71 times against an expected number of 
50; this makes a C.R. of 3.32, indicating, according to 
the usual normal approximation, a probability of 
0.00045. This probability should, of course, be multi- 
plied by 5 since there are 5 symbols. Nevertheless, 





20. The whole series of which the reproduced 10 runs for a part, 
was, from another viewpoint, examined by J. L. Kennedy, loc. 
cit. He also reached the conclusion that the shuffling was 
poor. 

21. The paper by Murphy and Taves (15) may, however, be mentioned 
as one of special methodological interest. These authors have 
devoted a careful study to the problem of adequacy:of shuffling 
and the problem of linkage in their experiments (cf. esp. pp. 
44 seq. and 62 seq.). Their results have, however, unfortunate- 
ly, no bearing on our present problem. As a matter of fact, 
the experiment in question differed essentially from the usual 
set-up in that several decks were used simultaneously; and the 
authors find that there was no significant linkage between di /f- 
ferent decks. The comparison of the distribution of a target 
deck against the succeeding one was a minor point: in this re- 
spect only the number of coincidences was counted and found to 
be within random limits. ‘This, however, is only a very special 
and unusual type of linkage. For the 10 runs of the MacFar- 
land experiment the number of coincidences is well within ran- 
dom limits: for the last 3 runs they are 3 and 6 respectively 
against an expected number of 5: yet we say that these three 
runs are interlinked. 


ne Te 
ee 


SEA RNS 
ee 


ha 








282 JOURNAL OF PARAPSYCHOLOGY 





es 


with the ordinary rule, odds of 0.00225 are still consiq. 
ered significant. In addition, there are other devia- 
tions: the expected number of occurrences of triples 
like ccc, etc., would be 2 for each symbol. Actually 
ccc occurs three times, sss once and the remaining thyge 
not at all. The explanation for this poor randomness 
is probably that the 250 decks, which were used, lay 
originally in the same order as they are delivered fros 
the factory; they were shuffled and cut, but this pro- 
cedure can never be relied upon to produce randomness, 


Of course, different methods of shuffling differ 
also with respect to their effectiveness. One of the 
methods consists in dividing the deck into several groups 
and reordering them: for this method the linkage can 
easily be shown to be very considerable. For this and 
many other methods there is not only the theoretical 
probability of a linkage; if the shuffler is not familiar 
with the dangers of shuffling it can easily happen that, 
in the consecutive removing and reinsertion of cards of 
the deck, more or less the same cards are affected, or, 
alternatively, that some group of cards is not affected 
at all. Both possibilities are rather probable because 
of the way in which the deck is held in the hand. The 
better methods of shuffling also often produce only an 
apparent random order. Consider, for example, the much 
advertised method in which the pack is divided into two 
nearly equal parts and the two halves are recombined in 
an alternating order, subject of course to small random 
fluctuations. The result may be schematically represent- 
ed by the following rearrangement of the original order: 


1, 14, 2, 15, 3, 16, 4, 17, 5, 18, 6, 19, 7, 20, 8, 21, 
9, 22, 10, 28, 11, 24, 12, 25, 13. 


This is of course but a scheme, and in reality sometines 
two cards might slip out or the two halves might vary in 
size. It is nevertheless clear that this procedure sep- 
arates more couples than a random rearrangement would 

or that one could safely bet to find the cards numbered 

23, 24, 25, in this order in the second half of the deck, 
If we repeat the procedure we might get another rearrange 
ment, e.g., 1, 20, 14, 8, 2, 21, 15, 9, °*°*°; again ram 
dom fluctuations will affect the order, but it is perfet 
ly clear that the effect cannot be a random order; there 


a 





22. I wish to make it perfectly clear that the above consideration 
does not intend to explain (or, indeed, analyze) Warner's ex- 
periment as such. Actually, his subject had no preference for 
the symbol c (having called c exactly 50 times). 
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are, e-g-, long odds that the cards numbered 24 and 25 
will still be, in this order, in the second half of the 
deck, that the cards 1 and 2 are separated by other cards, 
etc. The argument can be repeated, although naturally 
with each new shuffling the effect of random fluctuations 
increases until it, eventually, predominates. The ef- 
fectiveness of the method would be increased if the num- 
ber of shufflings were random; however, there if a natur- 
al tendency to use, consciously or unconsciously, always 
the same number of shufflings. 3 


5. In examining whether the shuffling effect is 
present in practice, each experigental series must neces- 
sarily be considered separately. Indeed, when combin- 
ing a larger amount of material, coming from different 
series, the ordering of the runs is arbitrary and the 
very notion of linkage loses its sense. Furthermore, when 
the whole material is submitted to random tests such as 
counting the frequency of pairs of symbols or of gaps be- 
tween them, the effect of possible poor local randomness 
disappears owing to the mutual cancellation of the dif- 
ferent methods of shuffling, the different initial ar- 
rangements, etc. (In some experiments the cards may have 
a tendency to cling together, as was the case in MacFar- 
land's experiment, in other experiments they may be sep- 
arated too frequently as was the case with the last meth- 
od of shuffling discussed above, etc.) In theory, of 
course, even poor local randomness can be tested on a 
large and heterogeneous material: but only by a study of 
the local variation of the standard deviation, which is 


~ ¥ laborious and requires a very large amount of materi- 
al, 


The possible effect of poor shuffling on ESP ex- 
periments has been frequently discussed with respect to 
the possible advantage the subject could derive from his 
knowledge of the card distribution in the previous ar- 
rangement. How, as has already been pointed out above, 
for the purposes of the present considerations, the ex- 
perimental set-ups are assumed to be the best: it may, 
e.g., be assumed that the subject gets no information 
about the actual card distributions, so that no predic- 
tions are possible unless ESP exists. However, the 





23. Thus, e.g., Woodruff and Pratt ( 16 , p. 130) explicitly state 
that they used 5 shufflings for each deck. 

%. Originally the following considerations were made during the 
discussion following the address when my attention was drawn to 
Greenwood's "chance control series" (6) which was said not to 
indicate any deviations from randomness. It may be mentioned 
that this series was intended, and is useful, for other purposes. 
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theory is based on a formula for the expected limits of 
the random fluctuations, and no mathematical formula cap 
be based solely on the lack of knowledge on the part of 
subjects. The main problem is whether the usual esti- 

mates for the expected number of hits (i.e., 5 per run 

of 25 trials) and for the standard deviation hold. 


For the first part the answer is affirmative, at 
least if it is assumed that the deck is properly cut, 
that is to say, if the deck is cut in such a way as to 
ensure that all 25 possible resulting arrangements are 
equally probable. This is true (and mathematically tri- 
viel) whatever the original distribution of the deck and 
whether or not the original distribution is known to the 
subject; and thig remains true even if the deck is not 
shuffled at all. #9) 


It is, however, perfectly clear that, with strict 
randomness not ensured, the standard deviation may be 
larger than the value usually assumed. To put it in 
plain language: The gross average of the number of hits 
for a large number of experiments with many subjects will 
not be affected by poor shuffling; but there is the pos- 
sibility that the mental habits of some subjects will, 
sometimes, cooperate with the shuffling effects so that, 
for some subject and some period, the average number of 





25. However, the assertion is not necessarily true if the deck is 
not properly cut. With several methods of shuffling the sub- 
ject may (consciously or unconsciously) derive a slight ad- 
vantage from the experimenter's habit of always cutting in the 
inner portion of the deck, that is to say, never to leave the 
deck as it was (as he is expected to do, on the average, once 
in 25 runs); in such cases a subject's habit not to call the 
same symbol at the same place way give him some advantage or 
disadvantage, as the case may be. 

I do not know whether this ever occurs, but the argument 
in itself is of interest. It must be clearly understood that 
if the chances of the subject exceed 5 correct guesses per rut, 
then, small as the difference may be, the subject is bound 
eventually to obtain "significant" deviations from supposed 
chance~average of 5. This argument was used by Kennedy, who, 
in certain experiments, discovered recording errors increasing 
the average score of 5 to 5.087. Assuming, which is plausible 
or at least possible, that the rate of recording errors is cor 
stant throughout the experiments, Kennedy rightly concluded 
that the recording errors can produce a "significant" deviation 
in 10,000 runs. Surprisingly, this argument was ridiculed by 
Stuart (27), p 309 seq. 
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hits will be sometimes above chance, and sometimes below. 
Thus the shuffling effect increases both the number of 
positive and of negative deviations from random expecta- 
tion, and the average remains unchanged. E.g., if the 
shuffling tends to reverse the order of cards (as was 

the case in Mac Farland's experiment) and hence increases 
the repetition of couples of symbols, this can help a 
subject inclined to call the same couples; and this ef- 
fect may be considerably augmented if the subject is 

told his results (in particular if the subject notices 
the occurrence of sequences of adjacent hits and tends 

to repeat such "lucky" combinations of symbols). 


It is clear that any kind of linkage between con- 
secutive target decks may (with more or less probability) 
get into such a correlation with the calls and affect 
the average number of hits favorably or adversely. Now 
in practice, if the scoring level is not above chance, 
then "there is, of course, no point in including these 
latter data in our efforts to compare ***. If average 
series are not above chance, there is nothing to say". 6 
With this we come again to the question of selection, 
raised in s2. It may be conceded that in each experiment 
the recorded totals include all results, so that there 
is no gprection of results in the individual experi- 
nents: the fact remains that, generally speaking, 
only experiments are published which are positive on the 
whole. And for this reason the practical effect of non- 
randomness of the card-distributions is to increase the 
relative frequency only of series with positive devia- 
tions. In theory it increases the probability for large 
deviations of both signs, but the selection effect of 
publication accentuates the increase in the positive de- 
Viations only and so produces a spurious increase in the 
average number of hits. 


Another serious effect of the linkage between 
consecutive target-decks lies in the possibility of pro- 
ducing a correlation between the scoring successes in 
consecutive runs; that is to say, under certain circun- 
stances, it may happen that a successful run increases 
the chance of success in the next. The full effect of 
such a correlation will become apparent first in the 
next paragraph, when we analyze the usual experimental 


26. Rhine (18), p. 181. 

27. This is true for the experiment by Rhine just mentioned, and 
I think, for all experiments of the last period. Admittedly, 
it was not always true. 
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practice, by which the experimenter may watch the suc- 
cesses, interrupt and start again with the experiment at 
will and even continue during the pauses the experiments 
"off record". If there is the slightest correlation be- 
tween consecutive runs, this procedure clearly amounts 
to a hidden selection of results. 


It might be objected that we are here considering 
exceptional cases not likely to occur frequently. And 
so we are. However, it should be borne in mind that we 
are also concerned with observations which are, on the 
whole, not frequent. The question is whether the poor 
randomness may sometimes produce some effect, or in 
other words, whether the assumptions underly}ng the 
usual ESP procedure are plausible or proved. 


Concluding these considerations I may mention 
the practice of some experimenters to check the subject's 
calls not only against the target deck but also against 
the reversed order of the latter. I assume that this is 
done for some special purposes; it should, however, be 
understood that this method has no bearing on the prob- 
lem of linkage or chance-correlation between consecutive 
runs, Indeed, it cannot be expected that a given series 
of calls should show an exceptional number of coinciden- 
ces both with the target decks and their reversed order, 
Thus a series with hits above chance expectation must in 
general be at chance level or less when checked against 
the reversed target. 


However, the whole problem of randomness could 
easily be avoided by a systematic use of tables of ran- 
dom numbers ( 11 and 29 ): the cards could thus be de- 
libergiely stacked in a random order unknown to the sub- 
ject. 


6. I proceed now to the question of so-called 
optional stopping. It is easily explained. If we de- 
cide to make, say, 1000 trials, there is a definite prob- 
ability 36 obtaining a C.R. of, say, 3. But until quite 
recently the corresponding formula was, wrongly, applied 





28. To forestall futile discussion I wish to emphasize once more 
that I am not discussing the existence of ESP and not proposing 
a counter hypothesis; generally speaking, however, I confess 
that I am unable to see any serious objection to the remark that 
a counter hypothesis is not applicable to all results. 

29. The same suggestion was made independently also in the address 
of Bartlett mentioned above ((3], p. 14). 

30. Cf. footnote 5. 
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to compute a quite different probability. It was not de- 
cided to make 1000 trials but the experiment was contin- 
ved until a favorable C.R. was obtained: if, in this 
way, the experiment was terminated, after, say, 1250 
trials with a C.R. of 3, the probability of obtaining a 
C.R. of 3 or more with exactly 1250 trials was computed, 
instead of the probability of getting it sometimes (with 
ring reasonable limits). In plain language: the probability 
i of a rainy day at Los Angeles may be negligible for ev- : 
6 ery day in the year: yet, by staying there, you are i 
sure, eventually, to get rain. 


its 
e~ 


SPREE 


The situation being as simple as this, it is sur- 
prising that the formula was misinterpreted; still more, 
that although this criticism was made by psychologists 
the validity 3f the argument was again and again vigor- 
ously denied. Thus it came about that a simple mathe- 
ot's matical fact was finally established by experiments rath- 
st er than by arguments: In [13] Leuba described a series 
of pure chance matchings, between two decks, conducted 





ee in accordance with the usual experimental set-up. He 
“ty made a total of 13,410 trials with 87 "subjects" and 
we found seven among them who did have, at one time or anoth 
a er, 200 consecutive trials with a C.R. of 2.5 or more. uf 
ae: One subject got 1000 trials with an average of 6.3 per . 
ar 25, giving a C.R. of 4.1 (according to the usual inter- 
in pretation the probability for this would be .00002). 
5t One subject even averaged 7.4 hits per 25 trials during 
250 trials and his average score over the whole experi- } 
ment (725 trials) was 6.3 (C.R. of 3.5). In the same ; 
d issue of the Journal of Parapsychology Greenwood [7] re- 
me ported on similar experiments, though conducted under 
des somewhat more favorable conditions. After these resu}}s 
re the effect of optional stopping was no longer denied. 
31. Today this statement is, in turn, denied, and it is instructive 
4 to quote one instance: (Greville (9), p. 251): 
7 "The statement is occasionally made that by continuing long 
rote enough one is certain eventually to obtain a fairly high average 
ite total score. This is clearly at variance with the recognized 
lied laws of probability, but in order to meet the objection more 
il specifically I have worked out a formula which gives, with a 
high degree of accuracy, the probability that a given score will 
‘ing ever be reached in a series of infinite length ***. For example, ' 
the probability in an infinite series of calls of ever securing, 
that at any time after the first 10,000 calls, a total average score 


of 5s correct guesses per 25 can be shown to be less than one 
SS in a million." 
In reality this probability is 1 (certainty) and one is 
sure to get, infinitely often, an arbitrarily large C.R. 
32. In the literature the theoretical discovery of the effect 
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Now the next question concerns the actual limits 
of the effect of optional stopping. It is certainly 
dangerous to belittle it with fallacious arguments. Thus 
Stuart ((28], p. 199) argues: "But in practice, it in- 
volves the same fallacy as occurs in gambling 'systems', 
A gambling system will work if an unlimited capital is 
available and an infinite number of plays are possible 
see," Now, the notion of "gambling system" is a matter 
of definition: according to the definition usual in the 
theory of probability no gambling system is possible no 
matter whether the capital or the number of plays be 
finite or infinite. On the other hand, there are long 
odds that a systematic use of the advantage of optional 
stopping will work satisfactorily. Indeed, this is an 
everyday experience, and many a 2. sdeagpencncaseal of the the- 
ory of probability rests upon it. 


The effect of optional stopping is certainly not 
capable of “explaining” record series like the one pro- 
duced by Riess, or indeed any series in which the subject 





(footnote continued) of optional stopping is ascribed to Green 
wood and Greville's [8]. As a matter of fact it is difficult to 
say when it was discovered since, in reality, we are only con- 
cerned with the correct interpretation of a formula which never 
before had been consistently misinterpreted. That in an infi- 
nite series any C.R. is bound to occur infinitely often is an 
immediate consequence of the very law in probability on which 
the whole ESP theory is based. A long series of mathematical 
investigations is devoted to refinements of this statement. Thus 
it is known since 1914 that in a series of n trials the order of 
magnitude of the greatest observed C.R. is expected to be | 
Vlog log n, and this quantity is steadily increasing. This gives 
an idea of what can be achieved in practice by stopping at favor 
able points. — Generally speaking, we are here concerned with 
a trivial special case of deep and interesting problems to which 
a considerable part of the modern theory of probability is de 
voted. 

33. Well known is the fact that many persons have succeeded in vis- 
iting Monte Carlo year after year and never losing. I myself 
have a friend who did it systematically, firmly believing in his 
ability of prediction. I never succeeded in convincing him that 
his experience was a common one. Indeed, if we suppose that he 
started gambling with a capital of $1000, he has obviously a 
considerable chance to gain $200 before ever losing the original 
capital. Stopping at this point, he has reasonable odds in favwr 
of repeating the same experience say, 10 times. There must be 
many persons with this experience. If we collect the data about 
them and combine into a single continuous series we again may 
get an arbitrarily large C.K. and refute the chance-hypothesis. 
No impractical long series are needed. 
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scored consistently above chance expectation, 34 It is 
even probable that the average effect of optional stop- 
ping (i.e., when the average is taken over the set of 

all tbinkpble experiments) would not be of decisive in- 
fluence. We are, however, concerned with a selected 
material, and the only question of practical interest 

can be: what is the probable effect of optional stopping 
at its best? We shall see that it fundamentally changes 
the aspect of a large portion of the experimental work 
produced until now. 


In (10) Greville gives a summary of his theoret- 
ical investigations on the frequencies of occurrence of 
different C.R.'s caused by the optional stopping. He 
reaches the conclusion (p. 90) that "these figures would 
indicate the adoption as a criterion of significance of 
a C.R. of 5% as roughly equivalent to the 2s which is 
customary under ordinary circumstances." And this es- 
timate leaves out of account the effect of linkage which 
at its worst may considerably increase the outer limits. 
Greville's estimate is the only ong I know of which is 
reached by mathematical deductions 6 and there it can 
certainly not be lowered if one wishes to keep on the 
safe side. 


Now a glance at the literature shows how funda- 
mentally Greville's estimate changes the whole aspect. 
It should be compared with the rule of the "Handbook for 
Testing Extra-Sensory Perception" by Stuart and Prett 
(p- 59): "If the C.R. is between 2.5 and 3.0, you can 
be relatively certain that the results did not happen by 
chance, If it is between 3.0 and 4.0, you can be very 
sure," It must be understood that, according to 





$4. Cf. footnote 28. r 

$5. Therefore no reliability can be attached to the numerical values 
of the effect as it occurred in the experiments of Leuba and 
Greenwood referred to above: they may be below chance and, at 
least, they cannot be assumed to be essentially above chance; 
hence they provide no limits. 

36. Greville's paper is not yet published (at least it was not at 
the time when Rhine's new book appeared).. I am unable to follow 
Greenwood's far more optimistic "definite but tentative recom 
mendation" (([7], pp. 225 and 229), which is reached on his em- 
pirical material. Ct. also footnote 35. 

37. In his table 7 referred to above ((1], pp. 95-97), Rhine sums up 
the main experiments "which acceptably exclude sensory cues." 
The table contains 34 entries with a total of 907,030 trials; of 
those only 14 entries with a total of 250,305 trials show a C.R. 
of at least 5s. Among those one single entry accounts for 











290 JOURNAL OF PARAPSYCHOLOGY 





Greville, a C.R. of 5¢ (i.e., 24 in the usual interpreta- 
tion), though commonly defined as "significant" must oc- 
cur fairly often. Indeed, according to the normal approx 
imation the probability of a C.R. exceeding 2.5 would be 
0.0062. A better approximation was computed by Stuart 
and Greenwood ([26], p. 303) noggr ding to whom the prob- 
ability in question is, roughly’, about 0.008, which 
means that because of optional stogging a C.R. of 5.5 
may occur once in 125 experiments. And, it must be in- 
sisted, this figure relates to the set of all started 
experiments: in practice, naturally, the experiments 
which turn out to have an unfavorable start are soon 
dropped, since they are unlikely to yield interesting re- 
sults. In view of the time and interest devoted to ESP 
at many American colleges it is certainly not surprising 
to har phat some experiments resulted in a C.R. of 

’ 


5.5. 


(Footnote continued) 96,700 trials. Not even all of these 14 
entries could make us feel "very sure" according to Greville's 
estimate combined with the Handbook's rule. 

38. Strictly speaking, this probability depends on the length of 
the experimental series. For a series of 625 trials Stuart and 
Greville find that the. probability is 0.0081 or 0.0093, re- 
spectively, according as the binomial or the matching hypothe- 
sis is accepted. For 2500 trials the corresponding figures are 
.0075 and .0082. 

39. The figure given is, of course, according to Greville, a bound 
rather than an estimate. On the other hand, his estimate leaves 
out of account at least the effect of linkage. 

40. When oe the various proposed recommendations as to 
which C.R. would be considered "significant" it is, unfortunate- 
ly, not always understood that, for special experimental designs 
the estimate is the more conservative the lower the chosen limit 
is. Such is the case when some items are tested as to the ex- 
pedience of their being subjected to further experimentation, 
= when those items are conserved which obtain a significant 

oR. 

41. It should.be noticed that there is also an effect of optional 
beginning parallel to the effect of optional stopping. As a 
typical example the report [2] may be quoted. The writer states 
that "***when beginning the experiments *** I kept no permanent 
records, merely recording runs whenever and wherever I found it 
convenient***. As the results began to surprise me, 2 began to 
use the record pad and made the observations at my house **’." 
With this favorable beginning the writer achieved a series of 
1000 trials with a C.R. of 3.44, which will no longer surprise. 
The next 1000 had a C.K. of 2.66, the third of .86 and the 
writer called "3500 additional cards with varying scores, at 
one time steadily below expectation’**." The writer then divides 
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However, even while accepting far lower limits of 
significance than a C.R. of 5.5 would be, we still can 
find reports of experiments reaching conclusions on ESP, 
and not including a single case of significance -- that 
is to say, based on purely chance material. Before quot- 
ing a couple of examples i have to mention the common 
practice of making experiments with several subjects, 
combining the obtained series into one large series and 
computing the corresponding C.R. as if this series were 
a continuous experimental series. It was already pointed 
out (s2) that if you stop each subject at a point slight- 
ly above chance (and you are practically sure that you 
can achieve this at will) you will eventually combine 
many series with absolutely insignificant but positive 
C.R. into one series with a highly "significant" C.R. Wg 
shall see that this is exactly what happens in practice. 


Now the same method is applied even in experiments 
with a sing}e subject. When the scores drop the experi- 
nenter may*’"'quit scoring for a time'. He may stop the 
testing for a minute, an hour, or a day °*** or he may 
suggest continuing the same procedure off record with 
the understanding that whatever scores are made, they are 
not to be part of the experiment. He may then watch for 
any trend of the off-record scores that may indicate a 
return to the previous scoring level. If such a trend 
appears, he may decide to continue the experimental tests 
for record." 





(Footnote continued) the last mentioned 4500 calls "into three 
groups (1) Morning calls when in good shape, (2) Evening calls, 
after work, ($) Morning calls when sleep had been lacking or I 
awoke with a sick headache". The last two groups slow negative 
C.R., the first 2.99. Such a division is, of course, practical- 
ly always possiblé and by no means significant. 

A related source of error was repeatedly criticized (cf. 
Warner (30), p. 92), and is now, I assume, completely abandoned. 
It consists in making so-called preliminary tests and continuing 
only with successful subjects but including the preliminary re- 
sults in the actual test made thereafter. Mathematically this 
changes the absolute probability of obtaining a specified C.R. 
into the conditional probability of obtaining it when it is 
known that the first few runs have been successful. The numer- 
ical difference is considerable. 

42, Against this obvious argument it has been objected that the 
combined series would not show a scoring level consistently 
above chance. This is equally true as beside the point if we 
are not informed about the total number of hits at each trial 
in the series: as we are not. 

43. Quoted from Stuart ((28], p. 200), where Rhine's method is 
defended against Kellogg. 
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Because of optional stopping this method is 
clearly unjustifiable since it means the addition of 
many small random errors (the point will be elucidated 
below by a simple mathematical example). However, in 
this case the method is obviously far more dangerous 
than it was in the previous example, because of the pos- 
sible effect of linkage. If there is any correlation 
between the number of hits in consecutive runs, this 
method is certainly most efficient in selecting positive 
results. 


To see how all these methods work in practice to 
create spurious significance let us, as a typical example 
consider Shulman's "Study of Card Guessing in Psychotic 
Subjects" [25]. The patients were grouped into 14 clin- 
ical classifications. The corresponding C.R. are 


0.53, 0.25, -0.37, <-0.58 -0.08, 0.09, -1.97, 
0.51, -0.14, 3.388, -1.937, 1.34, -1.62, -0.56 


which certainly yield an example of a random distributia, 
One group, however, the manic-depressive, have a "sig- 
nificant" C.R. of 3.388. Now we do not even need Gre- 
ville's rule that a C.R. of 5.5 should be considered as 
roughly equivalent to 2.5 and usual conditions to under- 
stand that the C.R. of 3.388 is not very surprising. It 
should also be noticed that we are here concerned with 
the probability that among 14 independent groups one at 
least should attain a C.R. of 3.388, and thus the rule 
of "significance" loses its foundation. The worst is 
that this C.R. of 3.388 is purely spurious. Indeed, the 
group consists of 12 patients and they attained the fol- 
lowing C.R. 


-0.49, 1.50, 0.90, 1.50, -0.39, 1.73, 
0.79, 1.19, 2.01, 1.45, 1.64, 1.56. 


This series does not contain anything extraordinary: it 
is a typical example of stopping each individual series 
at a moment slightly above chance and getting in this 
way a spurious C.R. for the whole series. The largest 
C.R., 2.01, was obtained in a series of 500 trials. 

Even without correction for optional stopping, the prob- 
ability of getting, in 500 trials, a C.R. of 2.01 or 
more exceeds .03: the probability that one among 12 
subjects gets such a C.R. exceeds 1/3 -- without any cor 
rections. Similar remarks hold for the other investiga 
tions, e.g., for the investigation on children by Louisé 
E. Rhine [17] which does not include a single case of any 
significance. 
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7. I may end by an example to show in simple but 
exact terms, what happens when several small series are 
combined into one. The example is not chosen so as to 
give any idea of the possible effect of the optional 
stopping: on the contrary, it simply neglects the effect 
of optional stopping altogether. The example is given 
because it is the simplest one I was able to find, and 
seems nevertheless to show clearly that tn principle the 
whole mathematical problem is changed when the technique 
described above is used. 


Let us consider a card guessing experiment, with 
independent trials, the probability of a hit being, in 
each trial, 1/5 (thus only the restriction to complete 
runs of 25 is removed for sake of simplicity). The gues- 
sing is done by the same subject. We adopt in advance 
the rule that the experiment shall be stopped when the 
subject has made a spwcified number, N, of hits. Thus 
there is no place for optional stopping -- the point of 
stopping may, but need not, be particularly favorable, 


Let, in an actual experiment, n be the number of 
trials before stopping. In order to compute the proba- 
bility of this event "N hits in n trials," we have, ac- 
cording to the common routine, to compute the quantity 
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We propose to show that this is not the true C.R., in 
other words, that the desired probability cannot be ob- 
tained by looking up the probability associated with X 
according to the normal law. 


It is convenient for the following to restate 
the problem in order to see its connection with our main 
problem. We may split up the whole series of n trials 
into N subseries, the first ending with the first hit, 
the next with the second hit, etc. With our convention 
when to stop it obviously no longer matters whether the 
guessing is done by one or by N subjects. The whole ex- 
periment consists of N tests: each of them consisting 
in letting a subject guess until he scored his first hit. 
Thus a "test" may consist of a single trial (probability 
for this = 1/5), of two trials (probability 4/5 . 1/5, 
etc. In other words, the length of a single test is a 
random variable: it is easily verified that its mean 
Value is 5, and its standard deviation 2V¥5. Hence the 
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total number of trials in N tests is again a random varj- 
able with the mean value 5N and the standard deviation 
2 J/5N. 


In this formulation the experimental set-up is 
strictly the classical one, and to calculate the proba- 
we = a certain pair (n,N) we have to compute the 

rue C.R. 


C= 2 J5N" 
It is seen that 
x» -c/2, 


so that X is not the true C.R. of the problem. To geta 
numerical illustration choose N = 180 and suppose that 
the whole experiment is terminated after n = 6000 trials, 
Then the true C.R. is C = -—5, whereas the usual procedure 
would lead to an estimate of X = 6.124. Similarly, ifn 
= 720, the usual method leads to a spurious C.R. of X = 
3.354 instead of the correct value -C = 7. For larger N 
the relative difference between -C and X decreases and in 
the limit it vanishes. Thus, in this case the numerical 
difference is not essential, but we have also agreed to 
choose an unfavorable rule of stopping and have abandoned 
optional stopping in order better to illustrate the the- 
oretical point. 


A more radical example of a similar character 
would consist in simply tossing a coin with the under- 
standing that the experiment has to end when the number 
of "heads" exceeds the number of "tails" by N.. The ex- 
periment can be restated as a making of N tests, each 
consisting in tossing a coin until the number of "heads" 
exceeds the number of "tails" by one. (This again means 
only splitting the whole series into N subseries.) Sup- 
pose the whole experiment takes n trials. To judge the 
probability of this event one would, according to the 
usual routine, proceed to calculate the "C.R.", namely, 


N -+ QN - n 
4n Vn 


Yet, in reality, the number n of trials in our experiwment 
is a random variable with infinite mean value: It is 
known that the probability of a given n, i.e.» of a cer 
tain X, can by no means be judged from the normal 
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probabilities. 44 Hence, though the experiment can eas- 
ily be conducted in practice, the "C.R." computed in the 
usual way is completely meaningless. 


To advocates of ESP these arguments may seem a 
reaction of too sophisticated a mathematical mind and 
the requirement to keep on the safe side in all esti- 
mates may sometimes appear hard. Yet it is only the ~ 
consequence of the decision to submit the ESP hypothesis 
to strictly statistical proof. If there is ESP, it seems 
natural to introduce optional stopping, etc., in order 
to allow for periods of fatigue and the like. Yet this 
is possible only at the price of introducing upper es- 
timates for the probabilities in question. On the other 
hand, one should bear in mind that the ESP ability must 
necessarily, though with diminished effect, become sta- 
tistically significant also if the right of optional 
stopping be given up altogether. 


Or else there remains the classical way which 
according to so experienced a statistician as R. A. Ftsh- 
er still seems more "relevant to the establishment of 
facts of nature": to find the most favorable condi- 
tions for ESP and to improve them until a reliable re- 


producibility of the phenomena under favorable conditions | 


is definitely proved. 


There is also a compromise between both methods, 
which has been frequently recommended: to substitute a 
deck of blank cards, or a deck consisting of identical 
symbols, for the usual one so as to change the frequency 
of occurrence of different symbols in a way unknown to 
the subject. This was declined by the authorities on 
ESP ([1], p. 17) on the ground that "this is to assume 
that ESP is a wholly conscious process and one that can- 
not be tricked". Nevertheless, it may be worth an experi- 
ment. For according to the same authorities, belief in 
ESP runs through all ages: but all ages tested it by 
ag to trick it. So, according to Shakespeare, when 

A 


Joan Pucelle came to see the Dauphin, he tried to 
trick her: 


"Go, call her in. -- But first, to try her skill, 





44. I omit the proof which is not quite simple. Despite the infi- 
nite wean value the law of large numbers holds (the early 
naive discussions about the so-called paradox of St. Peters- 
burg have been completely cleared up.) 

45. Cf. footnote 8. 

46. Cf. the quotation above and of si. 
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But 


1. 


10. 


11. 


Reignier, stand thou as Dauphin in my place: 

Question her proudly; let thy looks be stern: 

By this means shall we sound what skill she 
hath." 


she cannot be tricked: 


"Reignier, is't thou who thinkest to beguile me? 
Where is the Dauphin? Come, come from behind; 
I know thee well, though never seen before." 
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A REVIEW OF DR. FELLER'S CRITIQUE 
J. A. Greenwood and C. E. Stuart 


In the preceding article (5), Dr. Feller has 
raised three questions about ESP research: 


(1) Is the shuffling of the cards in ESP tests 
adequate to provide a random target series? 


(2) Does evaluation of published results, as a 
whole or in groups, constitute, because of supposedly 
unpublished chance findings, an improper selection of 
data for conclusions regarding ESP? 


(3) Has the ESP hypothesis been improperly favored 
by stopping experimental series at a suitable point? 


These are not new questions. Their familiarity 
in the literature of the ESP research is affirmed by the 
fact that they have been discussed by several of the 
major critics within the last six years. (20: pp. 414- 
415). The fact that they have been so discussed makes 
both the proper statement of the problems and the evi- 
dence bearing upon them readily accessible. 


We shall examine each of the issues to determine: 
(a) What ESP research results are exempt from the bear- 
ing of the question; and (b) What effect may be expected 
in research not exempted. 


Adequacy of Shuffling 


Statement of the Question. Is the hypothesis of 
inadequate shuffling of the cards in ESP tests adequate 
to account for the significant results reported? Inade- 
quate shuffling allows persistence of unbroken sequences 
in successive card orders. Under the hypothesis, these 
may coincide unduly with habitual sequences of the sub- 
ject's calls. Or the subject may be supposed to make his 
calls in patterns deliberately aimed at duplicating se- 
quences previously observed to recur among the cards. 





It will be shown here that (a) the major work 
bearing upon ESP is exempt from any nese bearing 
of the hypothesis of inadequate shuffling; (b) the 
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hypothesis is inadequate to account for even that minor 
section of the research to which it is relevant; and (c¢) 
the appearance of applicability of this hypothesis, as 
expressed by Dr. Feller, is contributed by spurious fea- 
tures of his demonstrations. 


Work That is Exempt. Much of the more important 
ESP research is clearly exempt, by virtue of the experi- 
mental conditions and the nature of the results, from 
explanation by the hypothesis of inadequate shuffling. 
Sufficient ESP work is conceded by Dr. Feller himself 
to be exempt to satisfy the needs of the ESP hypothesis, 
He agrees that: "It may be clearly stated that a few 
experimental series are recorded with so large deviations 
from chance that they obviously never could be reasonably 
‘explained' purely statistically. That is to say, no 
reasonable allowance for the effects discussed in the 
sequel would essentially change the aspect of those 
series. The two record series by Riess .... and by Mar- 
tin and Stribic are not even subjected to the effect of 
optional stopping ...." and "....I wish to make it per- 
fectly clear that the above consideration [lack of ran- 
domness] does not intend to explain .... Warner's ex- 
periment as such." 





there are other comparable series for which the 
grounds of exemption are equally good. The more obvious 
of these are listed in Section B of Table I, with the 
grounds for exemption noted in each case. 


Table I 
Work Exempted from Inadequate Shuffling Hypothesis 





A. Reports Agreed to be Exempted 











Reported by Runs Av. per Run C.R, 
Riess (23) 74 18.24 53.6 
Martin and Stribic, I (13) 1000 6.89 29.9 
Martin and Stribic, II (14) 1000 7.38 37.7 
Warner (28) 10 9.30 6.8 
B. Other Reports Obviously Exempted by Conditions or Controls* 
Rhine (Pearce-Pratt Series) (19) 74 7.53 10.9 
Rhine (PT) (19; 20: p. 416) 26 6.81 4.6 
Murphy and Taves (45) 2000 5.04 1.0 
Pratt and Woodruff (16) 2400 5.20 5.0 
Soal (26) Displacement Study 
Carington (4) Analysis of Drawings _ 





* The Pearce-Pratt series and the Pure Telepathy series are both 
high-scoring experiments in which cross-checks show no effective 
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. It is worth noting that every one of the series 


discussed in Chapter VI of Fxtra-Sensory Perception 
After Sixty Years as crucially supporting the ESP hypo- 
thesis falls into Table I, two of them with Dr. Feller’s 
agreement. It should be kept clear, therefore, that the 


‘discussion of this question from this point on is a 


largely academic one, so far as the status of the ESP 
research is concerned. 


The Question Answered. Although it is clear 
from the above table that the a prtori likelihood of in- 
adequate shuffling, as explaining any important research 
report, is small, the hypothesis itself may be dealt 
with in terms of its three aspects: (1) the theoretical 
expectation; (2) authoritative judgment; (3) and the 
evidence from empirical statistical checks. 





(1) That mathematically perfect randomness is 
theoretically impossible using card shuffling as a basis 
can be proved to be true. This proof, however, simply 
affirms the fact that no mathematical formulation can be 
rigorously paralleled in any set of actual observations. 
This completely skeptical attitude has always been ap- 
plicable but has never been useful in applied mathemat- 
ics. The closeness of approximation of a real situation 
to a mathematical hypothesis is customarily a matter of 
belief based upon trial and observation of the hypothe- 
Sis in question as applied. If the results are uniform- 
ly observed to be the same as if the hypothesis were 





(Footnote continued) degree of linkage. The Pratt and Woodruff 
series is a low-scoring experiment, but the precautions observed 
affirm the validity of the cross-check demonstration that no ef- 
fective degree of card~-linkage recurred. 

Of the Murphy and Taves work, Dr. Feller says: "These authors 
have devoted a careful study to the problem of adequacy of shuf- 
ans and the problem of linkage in their experiments..... Their 
results have, however,.... no bearing on our present problem." 
literally interpreted, this comment is contradictory since "a 
careful study" could scarcely yield results,of no bearing. However, 
the problem of linkage is shown to have no reasonable bearing upon 
their covariation results. 

Soal's experiments utilized card orders specially randomized 
from tables of numbers. Carington's drawings were randomized by 
& special technique of selection, the validity of which has never 
been challenged. 

The number of items in this table could be greatly increased 
by discussion of less obvious cases, but such elaboration is un- 
necessary since we are concerned here only with the demonstration 
that at least one significant research report may be excepted from 
the inadequate shuffling hypothesis. 
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actually satisfied, then we are content to use the the- 
oretical measures as a means of evaluating the practical 
results. Our confidence in so doing increases with the 
number of verifications which have accumulated, without 
any exception of extra-chance character. 


If poor shuffling is postulated, then there wil] 
be some kind of linkage observable between successive 
card orders in an experiment. This poor shuffling may 
result from mechanical features of the cards which fos- 
ter retention of certain orders and avoidance of others, 
If subject preferences accidently retain orders similar 
to those retained in the cards, the effect of this link- 
age will result in the variance of correspondences be- 
tween calls and cards being different from the variance 
of random expectation. Let us denote this as the hyp:- 
thesis of objective linkage. 


Another theoretical consideration is that shuf- 
fling techniques may foster retention of certain changes 
in order. The subject, if allowed to see the cards or 
the card records, might observe carefully the changes in 
previous card orders and thereby infer information about 
succeeding card orders. The effect of this, of course, 
would be to increase the number of correct correspond- 
ences. Let us denote this hypothesis as that of sub- 
jective linkage. 


(2) That shuffling is adequate to produce a prac- 
tical degree of randomness of certain types has been the 
considered conclusion of outstanding authorities in the 
mathematics of probability who have considered this 
problem. Emile Borel (2: p. 33) and Paul Levy (10: pp. 
48-49) have independently concurred that a reasonable 
number of shuffles of elementary types produce effective 
individual card randomness. That is, the probability 
that any specified card shall be in any specified place 
in the deck is 1/n, where n is the number of cards in 
the deck. This view would rule out the effective possi- 
bility of subjective linkage* and permit the problem to 
be wholly concerned with the hypothesis of objective 
linkage. 


(3) In order to meet directly the question of 
objective linkage it is necessary to know the empirical 
variance of ESP studies. The variance we need must be 





* Subjective linkage is, of course, not possible in those experi- 
ments in which the subject did not have any re of the caf: 
order or of past records (e.g., Riess (23), Warner (28), Rhine 
enclosed card series (17) ). 
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based on some aspect of the data which excludes possible 
changes in variance introduced by the hypothetical ESP 
factor itself. This variance is directly available from 
cross-check studies. If such patterns persist and card 
orders persist, then a cross-check involving the same 
population of cards and calls as the ESP test but with 
calls checked against card orders other than those in- 
tended will give the empirical variance we want. If ob- 
jective linkage is effectively present, this variance 
should be significantly different from expectation upon 
the hypothesis of randomness. 


Of the six crucial ESP series, the five which 
produced significant C.R.'s were cross-checked. None of 
these cross-checks gave a significant average, and none 
gave a variance significantly different from binomial 
expectation of 4.00. The results of these cross-check 
studies are listed in Table II. 








TABLE II 
Cross Check Data of Crucial Series (20: p. 43) 
: Method of ESP | Check} Check 
Wirhes Cross—Check pane Av. Av. | Variance 





Rhine (P.T.)|Calls vs. 3rd card 26 6.80} 4.88 | 3.64 53 
run preceding 


Rhine Calls vs. 3rd. 74 | 7.53| 5.20 | 4.81 .10 
(Pearce- card run follow 
Pratt ) ing 


Pratt and Calls vs. 3rd. (2400 5.20} 5.02 | 4.04 -37 
Woodruff card run follow 
ing 
Warner Calls vs. 3rd. 10 9.30} 5.60 | 4.04 34 
card run follow 
ing 

Riess Calls vs. 4th 74 | 18.24) 4.78 | 3.44 77 
card run follow 
ing 























The last two columns of Table II give the evi- 
dence relevant to the linkage question. Under "Check 
Variance" are the observed variances of the cross-checks. 
To find whether these were greater than chance expecta- 
tion, each was substituted in the formula 

x2 = Ns2 
o2 


wherein N is the number of runs, s? is the observed 
variance, and o2 is the theoretical variance of 4.00 
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expected on a binomial hypothesis. The last column 
gives the p-values of the x values so computed. The 
figures denote the probability that a random sample frog 
a population with variance 4.00 would give an observed 
variance greater than the one tabulated. If p were less 
than .01, the observed variance would be significantly 
greater than chance expectation. None of the p-values 
noted are significant. 


Although the variances appear quite different 
and are drawn from somewhat heterogeneous samples, a 
test of their homogeneity* yields a p-value slightly 
greater than .40, showing that the differences among 
the variances are attributable to chance variation. 


Since the variances observed are negligibly dif- 
ferent from those expected upon the hypothesis of ran- 
dom distribution, it may be concluded that any lack of 
randomness in the shuffled cards was not effective in 
any way upon the results. 


A relevant, but somewhat less crucial study 
which supports the results just stated has been made of 
the total chance check data of Table II of Fxtra-Sensory 
Perception After Sixty Years. The C.R. of each of the 
24 chance checks was found. The hypothesis was tested 
that each of them wis a random sample from a normal uni- 
verse with a mean of zero and a variance of one. This 
was done in two ways: forming a chi square by squaring 
the critical ratios, and summing with 24 degrees of 
freedom; and by using the approximate standard deviation 
of a standard deviation. Both methods yielded probabil- 
ittes greater than .01. If the variance applicable to 
these individual chance check series were appreciably 
greater than 4.00, then computing the individual C.R.'s 
using that variance should have resulted in a dispersion 
of the C.R.'s about zero significantly greater than the 
expected value of one. 


Dr. Feller's Position. Why, then, it is pertin- 
ent to ask, do the conclusions of the foregoing discus- 
sion vary so greatly from those implied by Dr. Feller? 
The main point of departure appears to be Dr. Feller's 
major attention to supposed illustrations of card link- 
age so superficially plausible that their actual bear- 
ing was never tested. Unfortunately, these illustrations 
were themselves misleading, as may be seen in Dr. Fel- 
ler's discussion of them. 





——— 





* Rider (22: p. 102) 
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Dr. Feller examines the symbol orders of a re- 
cord sheet from MacFarland's data (8). The 9th and 
10th runs of that record were as follows: 


(9) wrsws wpcss sperr cpprr wccwp 
(10) prrwc cwpcs sswpp rerwr swscp 


Considering the 9th order as numbered serially from 1 
to 25, he points out that it is possible to arrange 
those numbers to match the proper symbol in the 10th 
order in the following way: 


(9-10) 18,19,20,21,22  23,24,25,8,9 10,11,6,7,12 
14,13,15,1,2 3,4,5,16,17 


Of this Dr. Feller says: "It should be borne in mind 
that the deck was cut: thus after the shuffling proper 
the first card of the rearranged deck lay below the last 
one. In other words, the shuffling of the 9th arrange- 
ment of the deck left the group of cards numbered 16-25 
(i.e., 10 cards) in the same order; in addition, the 5 
cards numbered 1-5 clung together; likewise, the 4 cards 


In these remarks an inaccuracy of only one word 
occurs; namely, "cards" is used in place of the proper 
word "symbols." But this apparently trivial error ac- 
tually introduces a major assumption: that the order of 
numbers listed shows the true relation, card for card, 
between the 9th and the 10th target decks. 


Actually there are 24,883,200,000 possible orders 
that the cards might have taken to produce the order of 
symbols observed. The order of numbers Dr. Feller lists 
is a deliberately selected one of these possibilities. 
It is that order which would require the least reorder- 
ing of cards to produce the observed symbol order; or, 
in other words, that arrangement in which the most num- 
bers can be written consecutively in their customary 
order. But the a priori probability that this will be 
the true order is 1/24,883,200,000. And even if the 
cards were poorly shuffled, it would still be extremely 
improbable that the shuffling would produce the least 
possible disturbance of the cards. It is, therefore, 
virtually certain that Dr. Feller's analyses in no case 
presented the actual rearrangement of the cards.* 





* In fact the orders are, of course, deliberately selected to be 
those apparently most favorable to the inadequate shuffling hypo- 
thesis. 
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But if the number orders presented are false in- 
sofar as they represent the actual card orders, these 
deductions from them regarding shuffling or card linkage 
are completely erroneous no matter how convincing the 
deductions appear. Such statements as ".... the same 
groups of cards show a tendency to cling together...." 
and ".... it is probable that the deck was changed, or 
fell to the floor, between the fifth and sixth runs...," 
describe nothing that happened in the experiment. 


The lack of relevance of these deductive state- 
ments can be readily illustrated. We construct the fol- 
lowing successive ESP deck orders from Tippett's table 
of random numbers (27). (Twelve pairs of runs were ob- 
served.) In this case there is no shuffling or cutting, 
and there is no possible relation between the card or- 
ders. 


Order 1 prprs rerce wwwrp peswe wspss 
Order 2 wspws ppspr prsrc crwwe rscwc 


Although we know that there is no relation be- 
tween the cards in these series let us assume that Order 
2 is the result of Order 1 after shuffling and, following 
Dr. Feller's analysis, give Order 2 numbers, assuming, 
as he did, that the highest possible numbers of cards re- 
main together. 


(1-2) 21,22,23,13,24 15,16,25,1,2 3,4,5,6,7 9,8,11,12,10 
14,18,17,19,20 


On Dr. Feller's assumption that these numbers 
represent the true relation between the cards, we can re- 
construct easily the supposed act of shuffling which 
produced the second order. It is evident that the deck 
was finally cut between the seventeenth and eighteenth 
cards; the order before the cut being, therefore: 


1,2,3,4,5 6,7,9,8,11 12,10,14,18,17  19,20,21,22,23 
13, 24,15, 16,25 


It is evident, further, that the shuffling displaced 
only four cards, the 10th, the 13th, and the 15th and 
16th as a pair. Two other pairs (8 and 9; 17 and 18) 
were permuted without displacement. Since the shuffling 
disturbed only eight cards in the middle of the deck, we 
can deduce with confidence the actual shuffling act, 
which appears to have consisted of the single act of 
lifting the middle section of the deck loosely and slid- 
ing the lifted cards back into the deck with two or three 
motions. 
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We readily recognize these deductions as ridi- 
culous when we know that there was no shuffling and cut- 
ting of cards at all. But if these orders had occurred 
among ESP cards randomized by shuffling,there would be 
no such protection against the persuasive force of these 
deductions. We could easily, like Dr. Feller, agree 
that "after this example it can hardly be denied that 
linkages of the worst sort do occur." 


Dr. Feller was aware of the faults in his illus- 
trations, and he notes correctly that: "Strictly speak- 
ing, it cannot be logically proved that the cards actu- 
ally did stick together." But how, then, does an example 
show "that linkages of the worst sort do occur?" This 
contradiction is never resolved. It is evident, there- 
fore, that the approach does not provide a basis for 
rational judgment regarding the first of his three major 
questions. * 


Resumé. In answer to the question: "Is the 
shuffling of the cards in ESP tests adequate to provide 
a random target series?" it has been possible to show 
that shuffling is adequate within the degree of random- 
ness required. (a) The lack of randomness postulated in 
this que tion is not adequate to explain significant re- 
sults in numerous crucial ESP experiments. (b) The 
probable effect of lack of randomness, even if assumed 
to be present in some instances, is shown to be negli- 

ible. (c) The variation of these conclusions from those 
faplied by Dr. Feller evidently follows from his major 
dependence on spurious demonstrations of card linkage. 


Completeness of Publication of Results 


Statement of Problem. Is there a sampling error 
due to biased selection through supposed publication of 
only positive ESP results? If the results of all experi- 
ments are not published, may not those which do achieve 
publication be the favorable ones with positive conclu- 
sions? As e sample of "all ESP results," they would ac- 
cordingly constitute a biased sample. Any attempt to 
average or pool published results may, then, according 
to these hypotheses, lead to erroneous conclusions be- 
cause of improper selection. 





* It should be emphasized that we do not here even consider the 
problem of whether a significant degree or effect of linkage 
actually occurs in the MacFarland series (11). The statistical 
methods for studying linkage in the data are straightforward and 
well-known. Dr. Feller's analysis is not one of them 
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We will show that (a) the major work crucially 
bearing upon the ESP hypothesis is exempt from the hypo- 
thesis of improper sampling in publication, and, indeed, 
that all individual research reports are exempt; (b) 

. pooled results on which the question might bear have 
been sampled properly; and (c) that the criticism of 

Dr. Feller could only have arisen from erroneous supposi- 
tions regarding the pooling done. 


Work That Is Exempt. There has been no question 
raised as to the right of the individual investigator to 
report and evaluate his own work as a unit, nor is the 
right of the investigator to report his work in its ex- 
perimental subdivisions questioned. Hence, any adequate- 
ly reported individual research is exempt from the hypo- 
thesis of improper selection by incomplete publication. 
In fact, to list the individual research items properly 
exempt from criticism on this score would be to present 
almost the entire experimental literature on ESP. Ap- 
plication of this hypothesis is possible, then, only when 
results of different published experimental series are 
pooled or averaged. 





The Question Answered. Two kinds of pooling are 
possible and useful: (1) Pooling such as occurred in 
Rhine's original ESP monograph (18) and in Soal's recent 
total report (26). In each case, there is assembled the 
total result of all the several experiments of an indi- 
vidual investigator, work which may properly be conceived 
as constituting one large experimental project. It is 
assumed that he publishes all observations belonging to 
the project in question. 





(2) The second kind of pooling is that which oc- 
curs when results of various experiments having a common 
condition are summarized. This kind of pooling occurs 
in the tables of Fxtra-Sensory Perception After Sixty 
Years. These summaries have never been offered, however, 
as crucially requisite for proving the ESP hypothesis. 
(For example, the crucial evidence in the book mentioned, 
given in Chapter VI, is all in the form of individual 
research series.) The tables are obviously directed to 
answer a secondary question: What is the effect of com- 
bining all the available results of tests of a given 
type and condition, favorable and unfavorable? They per- 
mit, for example, the consideration of the effect of 
"negative series" cancelling deviations from positive 
work. A critical ratio from such a pooled table permits 
intelligent judgment upon these and other secondary ques- 
tions. This kind of pooling is directly within the 
bearing of the above hypothesis. 
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But even though (i) this type of pooling is not 
essential to the evaluation of ESP data and (ii) though 
it is not used except as an incidental means of keeping 
the total data in view, it is clearly a legitimate pro- 
cedure and may be defended in two ways: (1) It is a 
simple matter to determine the outer bound of work which 
could be supposed to exist unpublished, assuming it to 
have a chance average, without cancelling the signifi- 
cance of the deviations noted in a given table or pool. 
In fact, Table XII of Fxtra-Sensory Perception After 
Sixty Years gives the result of such a calculation. The 
amount of unpublished chance work necessary to reduce 
Table VII (the one attacked by Dr. Feller) would require 
170,000,000 trials. This is almost 200 times the amount 
of work reported in that table. It would be incontest- 
ably ridiculous to suppose that so much ESP research had 
ever been done, much less that so much that is worthy of 
scientific consideration could have been conducted with- 
out some knowledge of it reaching those interested in 
publication of ESP research. 


(2) A second answer to this question is that in 
the volume mentioned, Tables VIII-XI constitute the ad- 
mittedly most important summaries of ESP research. These 
summaries are made on the basis of the superior methods 
and precautions used. Since many of these have been 
developed in the Parapsychology Laboratory and since 
several of these precautions are of quite recent develop- 
ment, it is extremely unlikely that any completely in- 
dependent work belonging in those tables could be wholly 
unknown. For example, all of the research summarized 
in Table VIII was known in the Parapsychology Laboratory 
before it was published and no completed work meeting 
the conditions and known to the Parapsychology Labora- 
tory remained unpublished. There is, thus, every moral 
certainty that Tables VIJI-XI actually include all work 
done under the experimental conditions required for in- 
clusion. These tables constitute, therefore, permissible 
samples of ESP research and their significantly positive 
results cannot be attributed to selection of only posi- 
tive cases for publication. 


Dr. Feller's Position. Again, the difference be- 
tween these conclusions and those suggested by Dr. Feller 
should be accounted for if possible. We believe this can 
be done by pointing to the natural but erroneous supposi- 
tion of the critic that all experimenters with positive 
results hasten to publish and all with negative or chance 
results hesitate to do so. The erroneous character of 
this supposition is indicated in Fxtra-Sensory Perception 
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———— 


After Stxty Years, p. 75, where it was noted that of 
known and deliberately unpublished reports, thirteen had 
results favorable to the ESP hypothesis and seven had 
chance results. 


In the second place, Dr. Feller implies that 
the case of ESP is necessarily based upon pooled data, 
This is, as stated above, a mistaken assumption; in ESP 
research--as in all other research--the basic unit of 
evidence is the original research report. 


Resumé. In considering the question of sampling 
error due to biased selection in publication, the fol- 
lowing points are noted: (a) All the crucial ESP evi- 
.dence is exempt from the bearing of this hypothesis. 

) The hypothesis is shown to deal only with a secondary 
question: the right to summarize arithmetically the re- 
ports published. It is then shown that the summaries 
attacked are duly safeguarded against the source of er- 
ror implied in the hypothesis. (c) Dr. Feller's differ- 
ent conclusion appears to depend upon two erroneous sup- 
positions: that ye positive results are published and 
that the casé of ESP is based upon summarized data. 


Optional Stopping Effects 


Statement of the Question. Has the ESP hypothe- 
sis been improperly favored by stopping experimental 
series at a suitable point? The customary statistical 
formulas for significance assume that the stopping point 
in an experiment is independent of the statistical char- 
acteristic being measured. In ESP tests we know that 
chance fluctuations alone may occasionally produce runs 
of scoring above or below the expected chance average. 
The experimenter, by watching for these chance variations 
might, it is supposed, under the optional stopping hypo- 
thesis, produce consistently positive or consistently 
negative averages simply by stopping his experiment at 4 
favorable point. 





It will be shown here that (a) the major part of 
the work crucially bearing upon the ESP hypothesis is 
exempt from this hypothesis; (b) statistical methods have 
been devised to correct for this hypothesis and in prac- 
tice these corrections have been trivial; and (c) the 
apparent bearing of this hypothesis has been exaggerated 
by Dr. Feller and the correct criteria mis-applied. 


Work That is Exempt. There are several grounds 
for exempting work from the adverse bearing of the 
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optional stopping hypothesis. Not all of these need be 
invoked and discussed here. And it is not necessary to 
list all the exempt series, especially those requiring 
lengthy description of conditions, since the following 
erences of exception are ample to maintain the case 
for ESP. 


Dr. Feller's own concession that "the two record 
series by Riess .... and by Martin and Stribic are not 
even subjected to the effect of optional stopping...." 
permits exception of reports by 


1. Riess (23) 
2. Martin and Stribic, I (13) 
3. Martin and Stribic, II (14) 


Cases in which scoring rate was so high that the 
data remained positively significant at all points after 
the first few runs do not permit effective option. This 
requirement excludes reports by 


4. Rhine (Pearce-Pratt Series) (19) 
5. Woodruff and George (29) 
6. MacFarland and George (12) 


One experiment fully designed to limit option 
and to permit statistical correction for the limited 
option was the report by 


7. Pratt and Woodruff (16) 


Other series depending for their significance 
upon post-terminal analyses, and in which current know- 
ledge of the statistical item was unavailable, include 
those by 


8. Carington (4) 
9. Soal (26) 


Since, thus, there are major crucial experiments 
favoring the ESP hypothesis whose significance is not 
explainable by the optional stopping hypothesis, it is 
evident that optional stopping is not a necessary condi- 
tion for significance in ESP research. 


The Question Answered. Since the case for ESP 
is not involved in it, the actual technical issues of 
optional stopping may be viewed in a better perspective. 
There are two points to be cleared up at the outset: 
first, even among the ESP series not specifically exempt- 
ed above, there is none in which optional stopping is 
known to have been used as a matter of policy; that is, 
in which the experimenter has continued the experiment 
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to the point where it reached a significant C.R. and 
then stopped the experiment at that point. Ample evi- 
dence of this lies in the wide distribution of C.R's, 
very few of them being marginal (see, for example, Table 
XXIX, #xtra-Sensory Perception After Sixty Years). The 
question, therefore, must deal with the possible effect 
of stopping in spite of the fact that the attainment of 
a given C.R. probably seldom, if ever, actually deter- 
mined the stopping point. 


Second, the optional stopping hypothesis is rel- 
evant (if at all) only in regard to the point at which 
an experimental series, taken as a whole, is terminated; 
that is, the end point of the statistical data upon 
which the experimental conclusions are based. Thus, for 
example, if an experiment includes work with ten sub- 
divisions and the experimenter bases his conclusions 
upon the total work rather than upon the individual sub- 
series (e.g., work of individual subjects), then an op- 
tional stopping hypothesis may be applied only to the 
total results. The limits of individual subject per- 
formances are not stopping points from the viewpoint of 
the work as a whole and are accordingly irrelevant to 
the hypothesis. * 


Fortunately there are means of answering the 
question of the effect of optional stopping in ESP tests 
to an extent at least sufficient for present needs. 
There are two statistical techniques developed by which 
the maximum possible effect may be computed. Both of 
these are aimed at revising the probability obtained upon 
a simple sampling hypothesis to permit the assumption 
that option was used to the extent allowed by the methe 
ods. They require only a slight limitation in experi- 
mental procedure. 


The first method, suggested by Greville (7), ne 
cessitates the preassignment of the length of the pro- 
jected series to be between two rather wide-apart Limits, 
Greville gives three such possible categories of choice 
and with his formula could compute any desired number of 
them. The experimenter is enabled to select the largest 
C.R. obtained in the prescribed interval. The Greville 





* This may be easily seen if we consider a chance series to be made 
up by a number of throws of dice. Let us assume that we have a 
stock of ten pairs of true dice. For the chance series, it would 
be irrelevant if, for a time, one pair, and later another, were 
used to provide the data. A change of subject is the same as 
change of dice; in a chance series there could be no effect from 
option in such changes. 
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formula then gives an upper bound to the probability 

that by chance a C.R. as large or larger than the ob- ' 
served one could have been found in the prescribed in- \V/ 
terval of trials. This upper bound is not an approxima- 
tion to the true probability. And since the departure 
from the true probability is a function of the number of 
trials in the prescribed interval, it would be an ad- 
vantage to be able to decrease the size of the interval 

in the experiment. 


The practical difficulty of the restriction of 
interval led to another method suggested by Greenwood. 
The difficulty could be partly met by grouping the trials 
into as large blocks as convenient and treating the 
blocks as single trials. If the size of the blocks were 
determined at the outset, as well as an upper bound to 
the number of blocks, option would be possible, but so 
limited as to permit straightforward treatment. The 
value nP, wherein n is the upper bound to the number of 
blocks and P the probability obtained as of simple sam- 
pling, is an obvious upper bound to the true probability. 
In addition, if a little more restriction can be conveni- 
ently agreed upon by the experimenter (such as the se- 
cond block being half as large as the first) a more ac- 
curate approximation to the true probability can be 
found. * 


These methods are of recent development; so their 
application has been possible only in the most recent ESP 
research. The second method discussed was applied to 
the Pratt-Woodruff series (16). The probability of ob- 
taining the ennerver result was upon a simple sampling 
hypothesis, 3 x 107‘. The probability corrected for op- 
tional stopping was 5 x 10~%. It was thus possible to 
establish the fact that, assuming the full effect of op- 
tional stopping in that experiment, the significance of 
the result was not altered by the assumption. 


This application (though not its applicability) 
is unique in the field of ESP reseerch. We know of no 
other case in any field using the critical ratio method 
in which an experiment has been designed and evaluated 
to rule out the optional stopping hypothesis. 


Dr. Feller's Position. Dr. Feller's discussion 
of optional stopping is misleading because of his 


* The mathematical treatment of this method will be dealt with in 
full in an article devoted to general discussion of optional stop- 
2 and optional selection methods, soon to be presented for 
publication. 
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improper assumption that the probable effect of such 
stopping is known, whereas merely an upper bound is at 
present obtainable. This is evident in his treatment 
of Greville's formula. 


Feller quotes Greville's statement that with op- 
tional stopping a C.R. of 5.5 would be "roughly equi- 
valent" to the customary 2.5, but leaves to a footnote 
the content of Greville's very important qualification 
in his succeeding sentence: "It should be emphasized, 
however, that the probabilities given are merely upper 
bounds, not approximations." (7: p. 90) (Italics ours.) 


Thus Greville explicitly emphasizes the fact 
that his development dealt only with the establishment 
of a sufficient condition for elimination of optional 
stopping effects. Dr. Feller indicates that he is aware 
of this qualification; yet his discussion and applica- 
tion of the Greville criterion are based entirely upon 
the assumption that it is a necessary condition for sig- 
nificance. His inference that a C.R. of 5.5 has a 
chance probability of .008 is a specific elaboration of 
this mistaken assumption. 


The Greville formula does not give a chance prob- 
ability value to any C.R. either above or below the cri- 
terion, and Greville himself has indicated that it does 
not presuppose a chance explanation of C.R. values less 
than 5.5. All of Dr. Feller's uses of the Greville cri- 
terion make the latter improper assumption. 


A further instance of improper use of the option 
al stopping hypothesis occurs in Dr. Feller's attacks 
upon the validity of the Shulman (25) and the L. E. Rhine 
(21) experiments because of the probability of optionally 
stopping the work of the individual subjects. In neither 
report did the experimenters base their conclusions upon 
the assumption that the work of individual subjects was 
individually significant. Correction for the effect of 
optional stopping in these experiments must, therefore, 
be applied to the results which are claimed as signifi- 
cant; that is, the total pooled data of the subjects with 
L. E. Rhine (and the pooled data for each subdivision it 
Shulman's report). From the standpoint of optional stop- 
ping, since the series did not stop with individual sub- 
jects, the correction would be the same as if one subject 
had done all the work. 


Resumé. In regard to the question: Has the ES 


hypothesis been improperly favored by stopping experimental 
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series at a suitable point? we have noted: (a) That many 
important ESP series are exempt from this hypothesis. 
The possibility of optional stopping effect is therefore 
not a necessary condition for significant results in ESP 
research. (b) Statistical methods have been devised to 
correct for this hypothesis when the assumptions of 
simple sampling have been used. In practice these cor- 
rections have been trivial in their effect upon ESP re- 
search conclusions. It is reasonable to conclude, there- 
fore, that the probable effect of optional stopping in 
research series from which it is not excluded is not 
sufficient to explain the significant results reported, 
(c) That Dr. Feller's discussion of the hypothesis has 
exaggerated the probable effect because of false assump- 
tions leading to misapplication of the corrective cri- 
terion. 


Minor Criticisms 


In the foregoing we have been concerned with the 
three major questions raised by Dr. Feller's article. 
There remain a number of minor criticisms either speci- 
fically offered or clearly implied by his remarks. With 
the major issues resolved, these minor points are in 
many ways trivial, but attention to them is justified 
for the sake of completeness. 


(1) Dr. Feller's treatment of the question of 
biased sampling implies that ESP investigators exclude 
from report relevant data because the results are not 
above chance. 


This hypothesis is specifically dealt with in 
Extra-Sensory Perception After Sixty Years (pp. 118-121). 
The resumé here may therefore be brief and fragmentary: 
(i) Certain work (e.g., that of Pratt and Woodruff) was 
safeguarded against this possibility by the device of 
serially numbered record sheets: (ii) other series (e.g., 
that of Pearce-Pratt), by duplicate records independently 
preserved; (iii) still others (as that of Warner) had two 
experimenters present until the entire series (relatively 
short) was over and the total known; (iv) and finally, 
one such as the Riess series with scores so high that 
the loss of data could not have accounted for it in any 
Case. Many authors explicitly state that all the test 
results are included in the report. There is no good 
reason known for not accepting an author's statement on 
such matters. 
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(2) Dr. Feller insists that many events which 
are generally thought of as unusual are really not un- 
usual when the broad field of probabilities is consid- 
ered. Insofar as this point of view constitutes a gen- 
eral warning against too ready acceptance of an unusual 
event as significant of some unusual process, there can 
be no objection to emnhasis upon this aspect of prob- 
ability theory. It is necessary, however, to differenti- 
ate carefully the three implications of this view. The 
first implication---that probability theory never pro- 
vides basis for certainty of conclusion in research--is 
self-evident, as we have noted previously. The second-- 
that judgment of the degree to which an event is unusual 
should be based upon intelligently conceived probability 
theory--is sound scientific procedure. The third impli- 
cation--that since any probability, no matter how small, 
may be regarded as possible in the long run, it should 
not, therefore, be considered meaningful at all--is an 
unsound confusion of these first two implications. 


(3) The statement that "sometimes conclusions on 
the nature of ESP were drawn from material including no 
single significant case," implies that such a conclusion 
is improper. If so, this criticism would apply to all 
statistical analysis. ESP conclusions are always and ne- 
cessarily drawn from material that may be subdivided into 
units, no one of which is significant. A single call 


with a one-fifth probability can never be singly signifi- 


cant. Individual runs are seldom significant, even in 
highly significant experiments. Statistical methods are 
devised to derive conclusions from grouped data, a ne- 
cessity which would not arise if each individual datum 
were adequate to establish the conclusion. 


(4) Dr. Feller says that "ESP advocates have re- 
jected well-founded and almost obvious criticisms ... 
(specifically) the so-called effect of optional stopping, 
which has been pointed out by several critics." 


Lacking a specific reference by Dr. Feller, we 
can only say that we know of no ground for this statement, 
ESP advocates have, quite properly, rejected the optional 
stopping hypothesis as an acceptable counter-hypothesis 
for crucial ESP research, as we have done here. On the 
other hand, the question of the effect of optional stop- 
ping upon the critical ratios in ESP tests has been a 
specific research problem and the Parapsychology Labora- 
tory has openly fostered both discussion and research on 
this problem for several years. (3; 6; 9) 
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(5) Dr. Feller quotes M.S. Bartlett and R. A. 
Fisher as authorities expressing the need of caution in 
arriving at conclusions about ESP from statistical demon- 
strations. Review of the context (1) of the Bartlett re- 
mark reveals that he was emphasizing the difficulty of 
ascribing an observed significance to any real factor or 
process in an experiment. The Fisher quotation is simi- 
larly the affirmation of a general evaluative view in 
statistics--that consistent and repeated demonstrations 
of significance with reasonably small probabilities yield 
information of a more useful kind than a few observations 
of significance with extremely small probabilities. It 
should be noted that in both cases the writers are em- 
phasizing dependence of statistical interpretation upon 
the controls observed in experiments. In no sense are 
they challenging etther the validity of the statistical 
methods or the experimental results in ESP research. 
Fisher's approval of the soundness of the statistical 
methods was specifically given in 1935 (18B: p. 44). 


Of course the ESP research may revert to a com- 
mon sense evaluation by comparison of score averages and 
cross-check averages, and thus avoid the issue of statis- 
tical interpretation entirely. 


A fair test of the fruitfulness of a critical re- 
view is the amount of change necessary in contemporary 
methods of research to avoid the faults indicated. From 
our study of Dr. Feller's critique, it appears that no 
changes in method, either experimental or statistical, 
are required. 


Summary 


The three major issues raised in Dr. Feller's 
discussion concern the effect upon the conclusions of 
ESP research of (1) inadequate shuffling; (2) biased 
sampling from incomplete publication; and (3) optional 
stopping. 


We have shown that there is a considerable 
amount of experimental work manifestly exempt from ex- 
planation by these hypotheses. 


We have gone further to show that when each hy- 


‘pothesis is considered in terms of the experimental evi- 


dence on ESP, the probable effect in no case constitutes 
sufficient condition for the research results observed. 


It is our judgment that the chief fault of Dr. 
Feller's argument has arisen from his large preoccupa- 
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tion with plausible illustration (not the actual effect) 16 
of the hypotheses. This gave the misleading impression 
that the issues concerned the possibility of inadequate 
shuffling, biased sampling, and optional stopping. The 1" 
actual issues, the probable real effect of the hypothe- 
ses within ESP research, he left almost completely un- 16 
discussed. 
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A PERCEPTION RATIO STATISTIC FOR ESP TESTS 
A. A. Foster 


Experimenters reporting results of ESP tests 
have used gross results, average hits per run, and cri- 
tical ratios as illustrating extra-chance scoring. 

They have not, however, attempted to state quantitative- 
ly a measure of the extra-chance factor significant 
scoring purportedly reveals. 


A simple quantification of the perception in- 
volved in an ESP test is the number of symbols actually 
identified by the subject. These cannot be directly 
counted, but the number may be approximated by inferring 
from the total score how many symbols had to be identi- 
fied in order to attain that score. Let the total nun- 
ber of symbols or trials be N and the number of kinds of 
symbol and their mode of presentation be such that the 
expectancy of successfully naming a symbol by chance 
alone is p; i.e., the expectancy is pN if chance alone 
is operative. Now if X symbols are actually perceived, 
these X would be correctly identified and thus would 
constitute "hits." There remain N - X symbols which 
have not been perceived, and out of this number the 
chance expectation is p(N - X) hits. If H denotes the 
total number of hits, X the number of percepts, and C 
the number of hits actually occasioned by chance, we 


have 
H-eX-+C (1) 


The most probable value of C is p(N - X). The 
exact value, however, will be p' (N - X), wherein p' dif- 
fers from p by some small chance deviant. 


Equation (1) may be restated 
H = X + p' (N - X) (2) 


in which H and N can be observed. The equation solved 
for x becomes 


ite (3) 


From the Bernouilli Theorem it is deducible that 


for a sufficiently large N, provided N - X is also suf- 
ficiently large, the error of substituting p for ptis 
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negligible with relative certainty. With the theoreti- 
cally known p substituted for p!, equation (3) gives an 
estimate of x or the number of "percepts" that have oc- 
curred. 


: It is desirable to express the result as a ratio 
in order to facilitate comparisons between experiments, 





aii Mathematically it is simplest to use 

Perception Ratio “i or 

P.R. =H - pN_ 
: (1 - p)N (4) 
It is often convenient to state the result in 
ng terms of the percentage of the symbols which have been 
2 perceived. This suggests as a useful term the "ESP 
Quotient”: 

of 

ESP Q = 100 (H — pN) (5) 

(1 - p)N 

, The above formula may be derived by a somewhat 


different line of reasoning, as follows: Suppose H hits 
to have been made. The excess over expectancy is given 
by the relation H - pN. It is reasonable to suppose 
that, when N and N - X are large, at least that many per- 
cepts have been made. However, if H -— pN percepts have 
been made in such places as to add to the probable chance 
score, it is reasonable to suppose that percepts have oc- 
curred in similar ratio among those cases already select- 
ed by chance to be hits. To obtain a measure of percepts 
- we should therefore increase the excess of hits over ex- 
pectancy in the ratio of N to N - pN which gives 


X = H - pN a Ce 


N - pN 


X = tH as before. 


For the particular case of ESP cards, the value 


of p is 1/5; 
H - 4N 
BT 
hat + 
™ 
and equation (4) becomes 
H - IN 


P.R. = 5 (4a) 
aN 


(3a) 
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if h be used to denote average hits per run of twenty- 
five, we may write 


P.R. = hae (4b) 


and equation (5) becomes 
ESP Q = 5(h - 5) (5a) 


At this point it is imperative to consider the 
accuracy and applicability of the foregoing formulas 
within ESP test results. The doctrine of economy of 
hypotheses would bar their use in results which did not 
produce a deviation from chance expectation to a signi- 
ficant degree. That is, there must be in the data rela- 
tive certainty of the presence of an extra-chance fac- 
tor. On the other hand, however, if the measure is ap- 
plied only to significant series, it is not likely to be 
a true measure. The latter difficulty arises when, 
within a group of experiments, some series give sSig- 
nificant results and others give non-significant results, 
If enough of the series are significant, it is proper to 
postulate an extra-chance factor producing the devia- 
tions. But it is also reasonable to expect that of the 
expected chance deviations within the material the sig- 
nificant group would include somewhat more of the posi- 
tive cases than the non-significant group. For example, 
if the extra-chance factors were small and constant 
throughout the material, then the deviation produced 
might be added to chance positive deviations to make the 
total significant, but might be simply reduced to fall 
within the chance range if added to a chance negative 
deviation. For this particular problem the selection of 
only significant series for consideration is not a sta- 
tistically permissible sample. 


It must therefore be made explicit that use of 
the ESP Q should depend upon the validity of two assump- 
tions: (1) that ESP has occurred within the data under 
examination, and (2) that from the standpoint of a 
chance hypothesis this dataare a statistically permissi- 
ble sample. Under these circumstances the ESP Q is, 
while not an exact measure, the best approximation to the 
quantitative effect of a postulated ESP process. 


A further question is that of reliability of the 
ESP Q. The statistical constant needed is the standard 
deviation. Before presenting any such measure in this 
case it is important to stress the fact that ordinarily 
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two rather different variates are collectively described 
by a single S.D. For example, it might be stated that 
the mean weight of a certain animal was 20 gms.; S.D., 5. 
This S.D. is almost wholly a measure of the range of 

size for the creature concerned since the actual weight 
at any one measurement could be determined within what 
are comparatively very narrow limits. On the other hand, 
when the atomic weight of an element is quoted at 1.00; 
S.D., .0013, it is tacitly assumed that the atomic weight 
is a constant and that the different values found for it 
represent experimental error only. Or to cite an inter- 
mediate case, a psychologist might, after employing sev- 
eral tests, state the 1.Q. of a certain group to be 105; 
§.D., 12. Here the &.D. would express both the inherent 
variance within the group and the uncertainty attending 
the use of the tests employed. 


Any S.D. associated with an ESP Q would be most 
comparable to the last instance cited above with the ad- 
dition of certain complicating factors peculiar to the 
type of test conducted. Since a major use of S.D. is 
to afford a measure of significance of difference in ex- 
perimental results by noting the ratio 


M - Mz 


Vo? + oF 


it would be necessary to be sure that the means compared 
were actually comparable so that their corresponding 
8.D.'s are likewise comparable. For example, suppose we 
have 


Group Technique ESP Q Ss. D. 
A x Q, Oy 
A y Qe Oe 
B x Qs Os 
B y Q., O4 


Q:Qc, QiQs, Q.Q,, and Q.Q, are comparable; whereas Q,Q, 
and Q.Q, are not unless control groups be built up. 
Several different features might be illustrated by the 
wse of an ESP Q and its S.D. Suppose, for instance, an 
experimenter had had A subjects make B runs of twenty- 
five cards each. It would be possible to state: 


1. That the ESP Q of a given subject for that 
experiment was so and so with a S.D. based on the root 
mean square deviation of the ESP Q values indicated by 
the successive h value which that subject actually 
achieved. 
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2. The ESP Q of the group might be given with 
a S.D. compounded of the sigmas of the individual sub- 
jects. 


3. Another investigator might group the ulti- 
mate findings above with those of other experiments and 
arrive at a grand mean ESP Q with its associated S.D. 
computed from the o values of the constituent quotients, 
Obviously, their categorical limits would have to be 
strictly reckoned with in any comparison of values. 


"Low-aim" scores in which the subject attempts 
to avoid “hits" may be evaluated similarly to the custon- 
ary method. For low-aim, the p value for ESP cards be- 
comes 1 - 1/5 and formula (4b) becomes 


P.R. 7254 h (4c) 


This formula is identical with one previously and inde- 
pendently derived by Prof. R. H. Thouless. (Vide "Dr. 
Rhine's Recent Experiment on Telepathy and Clairvoyance 
and a Reconsideration of J. E. Coover's Conclusion on 
Telepathy," Proc. Soc. for Psy. Research, Vol. XLIII, 
1935, pp. 24, 32, 37.) 


Use of the ESP Q may be exemplified in the fol- 
lowing problem. Before the use of ESP cards, frequent- 
ly used bases for ESP tests were the common numbers fron 
zero to nine. The question arises whether a chance prob- 
ability of 1/10 in the test material is better than that 
of 1/5. It was noted that from all results reported of 
tests utilizing the 1/10 chance basis a total of 34,078 
trials gave 3,985 hits. All tests using 1/5 chance basis 
amounted to 2,758,354 trials with 604,403 hits. Sub- 
stitution in formula (5) gives ESP Q's of 1.9 and 2.4 
respectively. Although the reliability of the values 
is unknown, the difference suggests the superiority, in 
terms of the number of items "perceived," of a 1/5 prob- 
ability in the test material. 





1. “Extra-Sensory Perception after Sixty Years," p. 271. 
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IS ESP DIAMETRIC? 
By A. A. Foster 


When the ESP test is like that of Blind Matching, 
wherein two unknown cards are matched in an attempt to 
produce correct matchings, the perception may consist of 
the identification of the two cards followed by an infer- 
ence that they belong together, or the perception may in- 
volve the direct experience of "similarity" without nec- 
essarily including identification of elements adjudged 
similar. The former possibility I shall call the "cir- 
cumferential" hypothesis; the latter possibility, the 
"diametric" hypothesis. 


Let the probability of success in a given opera- 
tion be po on a basis of pure chance. Let the probabil- 
ity of non-success be qo. In N trials the expectation 
is poN = Ho. Suppose actual performance results in H, 
successes or hits. Assuming that the ratio H/N is rea- 
sonably stable for large N, let us denote the value 
which this approaches as p,. 


Let us assume that p, which is of the nature of 
an empirical probability (such as a death rate) is the 
result of joint operation of two factors. One of these 
factors we call chance and denote its rate of production 
of successes by p.. The other factor wewmight call ESP; 
and denote its rate of incidence by p,. Let us denote 
the chance event associated with p,. by E, and the ESP 
event associated with p. by E.; let E, denote an event 
regardless of its association. It is possible to con- 
ceive situations where an an occurrence of an E, is oc- 
casioned by an occurrence of 


(a) Eo or E,; then p, = po + po 

(b) E, and/or E,; then p, = po + Pe — Pode 
(c) Eo and Eg; then py = Pode 

(d) E, and not E,; then p, = pods 

(e) E, and not E,; then p, = dope 


In general p, = f(po,P2) wherein, as previously 
defined, p. is the pure chance probability of incidence, 
and p. is the rate of incidence of a perturbing factor, 
possibly ESP. 
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If we consider the case of simple matching of a 
deck of ESP cards to five visible key cards (the Open 
Matching Procedure), we have the relation 


Ps = Po * Po- PoPe (1) 


since perception and/or chance may occasion a hit. But, 
as noted above, H,/T as an observed rate may be consid- 
ered a measure of p,. Therefore for ESP cards we have 


Hs = (1/5 + pa - 1/5pe) (2) 
whence, for a run of 25 
H,sh- 5 (3) 
T 20 


where h is the number of hits or successes in the run. 
(This is, of course, the ESP Q which was defined and de- 
rived by other means in a previous article.) 


We will now consider the somewhat more complicat 
ed procedure of endeavoring to match cards from a deck 
to five key cards which are themselves known sensorially 
(the Blind Matching procedure). If ESP were to.follow 
what we ordinarily call logical steps, the process would 
be as follows: An attempt would be made to envisage the 
key cards and then an effort made to envisage the deck 
cards and place them opposite the assumed corresponding 
key cards. The probability of a chance success in the 
first operation we will denote by p,, and that for the 
second operation we will denote by pp, and for the con- 
plete operation p,,,- For ESP cards we have the chance 
expectation 

Pox * Pop * Popm * 1/5 


Let the rates of incidence of the corresponding perturbd- 
ing factors be poy, Pop and Popy and let the corresponding 
rate of incidence be pix, Pip, and Pipy- Let the corre- 
sponding items for open matching be Doom * 1/5, Pdy and 
Piom: A priori one would expect Pooy, Pox, and Pop to be 
of the same order of magnitude. Now if either pp, or 

Pop be zero, Pipy will tend towards 1/5 as a limit no 
matter how high the other of the above factors might be. 
Further, a perfect score over a long series could only 
be obtained if both p,, and pop approximated very close- 
ly to unity. 
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In practice the values of poy and Pop, Or even 
Pix and pip are not calculable from the records. How- 
ever, if Poy and p exist at all, their product is pp,,y. 
This follows from the fact that if we postulate ESP to 
take place in steps, Posy is dependent upon coincidences 
of pbx 2nd Pop: 


p and pop are of the same order as p.o,- There- 
fore, if £Sp takes place by logical steps, it is to be 
expected that p.,, is of the order of (poo,)*. 


Actual experimental results indicate that po.y 
is much closer to pooy than to (poo,)*. Rhine and as- 
sociates found that in total comparable OM_and BM series, 
the ESP Q's were 7.2 and 2.6 respectively.+ A circun- 
ferential hypothesis would predict the BM results to be 
of the order of ESP Q = 0.5. These suggest that the ESP 
function proceeds diametrically to its end quite inde- 
pendently of the ordinary circumferential steps of logic. 
Or, in other words, at levels of action ordinarily asso- 
ciated with ESP experiments, little if anything is posi- 
tively cognized; the perception is rather, as far as we 
know, one of correct or incorrect response. 





To facilitate further research into the question 
of diametric or circumferential action of ESP, the fol- 
lowing types of experiment are proposed: 


1. A group of subjects should try: 


(a) OM 

(b) BM 

(c) AOM, or Arbitrary Open Matching, in which 
& success would consist, not in matching like to like but 
in associating the cards according to an arbitrarily pre- 
determined but unrevealed code. For example, the pairs 
whose association would be termed a hit might be, for ex- 
ample, (OL), (A) , @+), (LA), and (+0). 

(d) ABM, or Arbitrary Blind Matching, which 
“sg differ from AOM only in that the key cards are face 
own. 


Two sub-conditions of this experiment suggest 
themselves. In one case, let the arbitrary code remain 
fixed throughout and in the other let it ers after 
every run. An ESP machine would greatly facilitate this 
action. The particular point in introducing procedures 
(c) and (d) is that on a circumferential hypothesis, the 
ESP Q to be expected from them is of the order (p},)° 
which would usually be very small indeed. 





1. Extra-Sensory Perception after Sixty Years; p. 318. 
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2. <A group of subjects should try: 


(a) calling whether a die of six faces 
showed an odd or an even number, 


(b) calling odd or even according to the 
major number of such faces presented by five dice rolled 
at once. (We must employ the number of odd and even 
faces out of a total odd number rather than the sum of 
their face indications, since the probabilities for odd 
and even sums are different.) 


If ESP is diametric, both case (a) and (b) are 
simple dichotomous decisions between odd and even. Hovw- 
ever, if ESP be circumferential, the task of envisaging 
five dice would seem at least five times as difficult as 
envisaging one, and the score attained should reflect 
this difficulty. 
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A CRITICISM OF DR. PRATT'S USE OF CHAPMAN'S 
"STATISTICS OF THE METHOD OF CORRECT MATCHINGS" 
IN THE EVALUATION OF ESP IN DRAWINGS? 


Douglas G. Ellson 
Stanford University 


In 1937, J. G. Pratt published an analysis of 
the data obtained in a series of experiments on extra- 
sensory perception by the late Dr. C. Hilton Rice (2). 
A part of these data consisted of two sets of 64 draw- 
ings obtained in informal ESP experiments. A number 
of "agents" had been used, working over a period of 
weeks. The agent, seated alone in a room, made an 
original sketch (0) of some simple object or design. 
At the same time, Dr. Rice in another room and without 
knowledge of the 0 attempted to sketch a reproduction 
(R) of the original. If similarities between members 
of the 64 pairs of drawings obtained in this way were 
more frequent than could be expected by chance, the 
operation of ESP would be indicated. 


After discussing earlier methods of evaluating 
the presence of ESP in investigations using drawings, 
Pratt says: 


"There is a more direct method of evaluating such 
work. This is a procedure called the method of correct 
matchings, which has an established statistical basis. 
The method is one which has been widely used in psycho- 
logical work, but this is its first application in a 
parapsychological study." (2, pp. 251-252) 


It is the writer's opinion that Pratt's interpretation 
of the statistic which Chapman (1) presents for use 
with the method of correct matchings is incorrect. 





1. This paper has been accepted and approved by the Stanford Com 
mittee on Psychical Research as Communication No. 9 from the 
Psychical Research Laboratory. 

2. This assumes that sources of communication (other than ESP) 
between the two persons taking part in the experiment are ex- 
cluded. Pratt indicates that this latter condition is not met 
satisfactorily in the Rice experiment, and consequently does 
not interpret the extra-chance similarity which he obtains by 
the use of Chapman's method as evidence for ESP. 
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First, let us examine Chapman's contribution 
(1). It presents a statistical basis for evaluating 
the results of experiments in which a group of judges 
attempt to match two sets of paired items. An illus- 
tration will make clear the application and interpreta- 
tion of the statistic: 


Four judges are given 10 samples of handwriting and 
10 character sketches, each sketch describing one of the 
persons whose handwriting is represented. The judges 
attempt to match each sample of writing with the corre- 
sponding description of the writer, and succeed in match- 
ing correctly 4, 4, 5, and 7 pairs respectively. What is 
the expectation that they would do this by chance, i.e., 
that 4 judges would average 5 correct matchings? 


Chapman presents a table (1, Table VII, pp. 
296-297) which he describes as follows: It "gives the 
probability (p) of obtaining a mean number of correct 
matchings as great as 8s or greater from n independent 
trials ee from n judges or S's) when the number of 
things to be matched (t) in each series is four or 
more." Thus we have: 


p = the probability of obtaining a mean number 
of correct matchings as great as s or greater 


s = the mean number of correct matchings 
n = the number of judges 
t 


= the number of things in each series (must be 
4 or more for the table to be applicable) 


In the above example, n = 4, 8 = 5, and t = 10. 
With these values, we find from Chapman's tables that 
p <.001, that is, the probability of s being as great 
as five is less than 1 in 1000. 


We may conclude from this small value of p that 
the judging was not done on a chance basis, i.e., that 
presumably the judges were able to recognize some re- 
lationships between corresponding members of pairs and 
- re a0 handwriting samples with the correct character 
sketches. 


It is this statistic which Pratt applies to the 
Rice data. His treatment is best seen from a direct 
quotation: 


"All of the 0's and R's were traced on two sets of 64 slips 
of paper each (small scratch-pad size). Each set was then 
shuffled thoroughly and the random order of slips in each 
set numbered from 1 to 64. 
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As the method of correct matchings had never before been 
used for results of this type, it seemed desirable to find 
the results of matching different numbers of slips in a 
group. ‘This was done in order to get a basis for estimat- 
ing the reliability of the method and to see how different 
sizes of matching units support further analyses of the re- 
sults. Three sizes of matching units were tried with the 
following results: 


"(1) Four judges were asked to match all 64 of the R's to 
the 64 0's. The successes were 2, 4, 4, and 5, giving an 
average of 3.75 where a value close to 1 would be most ex- 
pected by chance. From the tables which Chapman gives, we 
find that the probability that this would occur by chance 
is less than 1 in 1,000 (this is as far as the tables go), 
which justifies assuming some principle tending for cor 
rect matchings. 


(2) The above 4 judges and 2 additional ones were asked to 
match 16 groups of 4 R's each to 16 such groups of 0's. 

In each matching unit, the 0's and R's belonged together, 
but the judges had no clues to the correct matchings ex- 
cept those intrinsic to the drawings themselves. 


Each of the 6 judges had 16 separate matching groups, mak- 
ing 96 separate matching groups in all. The number cor- 
rectly matched was 174, or the average number of correct 
matchings per group of 4 pairs was 1.81. By consulting the 
same tables (Chapman, ibid.) as before, we find that the 
probability that this would occur by chance is again less 
than 1 in 1,000 and is therefore highly significant. 


(3) The 6 judges who acted in the second matching condition 
and 3 additional ones were asked to judge in 32 matching 
units consisting of 20's and 2 R's each. The 4 drawings 
making up each group had been obtained together in two 
tests. The object of having them matched by the judges was 
to see whether an R tended to be more similar to the 0 of 
the same test than to one other 0 chosen at random. In 
each of the 32 matching units, a judge had to be right for 
both pairs or wrong for both, with a chance probability of 
1/2 for either result. The results may be considered, 





8. In forming these sub-groups, the R's were taken by groups of 4 
as they appeared in the shuffled set of slips, and the corres- 
ponding 0's were selected from the other shuffled set of slips 
and presented in the order in which they were found. For ex- 
ample, the first matching unit was presented as follows: 

ip ne | 2 3 4 

R's 2 43 49 54 
For a perfect score, the R's would need to be rearranged as 
follows: 

s.. 2 2 3 4 


“eo ts 3B SS 
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therefore, on the basis of 32 trials for each judge with 
an average chance expectation of 16 successes. The actual 
number of units in which correct judgments were given were 
21, 20, 19, 18, 24, 21, 21, 23, and 22. This gives a de- 
viation of 45 beyond mean chance expectancy, which wopid on 
the average happen less than once in a million times. 


The three applications here made of the method of correct 
matchings (64 with 64, 4 with 4, and 2 with 2) all give 
highly significant results." 


Pratt then interprets the probability values ob- 
tained by these procedures from Chapman's tables as in- 
dicated by the following: 


"Thus far we have been concerned only with finding and ap- 
plying a suitable method for determining whether there is 
anything more than a chance similarity between agent's and 
subject's drawings in the 64 tests. he method applied shows 
that there is a significant degree of resemblance, and this 
fact raises the second question of how to account for the 
results, Was the more-than-chance relation shown in these 
drawings due to ESP?" (2, p. 254, italics ours). 


It would appear that two criticisms may be made 
of the treatment indicated in the quotations given 
above: (a) the interpretation of Chapman's tables in 
terms of resemblances between the two series of draw- 
ings is not justified, and (b) Pratt uses an inflated 
n in the calculation of his critical ratio. Each of 
these criticisms will be discussed in detail below. 


(a) The interpretation of Chapman's tables in 
terms of resemblances between the two sets of drawings 
is not justified. 


Unfortunately, it is not possible to quote 
passages directly from Chapman to demonstrate this 
point. It is evident that he did not foresee the mis- 
interpretation which Pratt makes. However, it can be 
seen that Chapman's statement that his table "gives the 
probability (p) of obtaining a mean number of correct 
matchings as great as s or greater" is very different 
from Pratt's statement that "the method applied shows 
that there is a significant degree of resemblance." 

A statement concerning reliability of a judgment is in- 
terpreted as a statement of the probability of occurrence 





9. The formula used is C.R. = dev.//npq where n (trials) = 288, 
p (the chance of a success on each trial) = 5 and q (the chance 
of a failure on each) = 3. The C.R. is 5.3 which gives the 
odds against chance stated. 
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of the events judged. Perhaps the clearest way of demon- 
strating the difference is by the use of an illustration 
in which the probabilities relevant to the events to be 
judged are known independently. 


Let us suppose that two subjects each make five 
sketches, and that both subjects were told that only 
the five ESP card symbols were to be sketched. These 
subjects draw their sketches in the following order: 





Trial Subject I Subject II 
hs Circle Circle 
2 Plus Waves 
3* Star Star 
oe Square Square 
5 Waves Plus 


Starred trials are "hits," i.e., the drawings are simi- 
lar. The probability of three hits (pairs of similar 
drawings) being sketched on the same trials may be conm- 
puted directly by means of conventionai probability for- 
mulas. Three hits in five trials would be expected ap- 
proximately once in 11 times. 


If the sketches made by the two subjects are now 
given to five judges who were told to match drawings in 
one series with similar drawings in the other, we should 
expect all five judges to return the drawings in the 
following pairs (not necessarily in this order): 


Circle. eee -Circle* 
Star. eeeee »Star* 
Plus.. eeee -Plus 
Square.....Square* 
Waves......Waves 


Each of the judges would have matched correctly the 
starred pairs. The average number of correct matchings 
for the five judges would thus be 3.00. Looking in 
Chapman's table under n = 5 and s = 3.00 (t being great- 
er than 4), we find p <.001. From this we should con- 
clude that the judges were not matching on the basis 

of chance alone, since if they had done so, they would 
have averaged 3.00 or more correct matchings less than 
once in 1000 times. We might extend this conclusion to 
say that the judges could recognize similarities be- 
tween drawings in the two series. We could not, how- 
ever, interpret this as Pratt has done in his parallel 
case, and conclude that the number of similar drawings 
made on the same trial by the two subjects’ would be 


3. It is assumed from context that this is what Pratt means by 
the term “degree of resemblance" between the two series. 
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expected less than once in 1,000 times, since we have 
already shown that this would occur once in 11 times. 


This probability (1 in 11) represents the 
chance expectation that Subject II should sketch three 
or more of the five drawings on the same trials as Sub- 
ject I, or that one judge should make three or more 
correct matchings. Obviously the chances are far less 
for two or more judges to make three correct matchings 
(i.e., to average 3.00). Pratt's interpretation as- 
sumes the probabilities in the two cases to be the same, 


(b) Pratt uses an inflated n in the calculation 
of his critical ratio. 


At the end of his description of his third ap- 
plication of Chapman's statistic (2, p. 253, see quote 
above) Pratt states that a (total) deviation of 45 cor- 
rect matchings obtained by 9 judges would happen by 
chance less than once in a million times. This prob- 
ability figure is obtained from a critical ratio of 
5.3, which is given, together with the figures on which 
it is based, in his footnote 9. The critical ratio is 
computed by means of the formula: 


C.R. = dev.//npq 
where dev. = 45, n (trials) = 288, p = s, and q = $. 


Two of the figures used in this computation may 
be questioned. The deviation (45) is the total number 
of correct matchings made by all 9 judges in excess of 
chance expectation. The nm is obtained by multiplying 
the number of matching units (32) by the number of 
judges (9), which gives a total of 288 trials. 


The use of an inflated n will be obvious at 
once to the majority of readers. However, for those 
who are not familiar with the effect of this statisti- 
cal error, a demonstration may be made by submitting 
our hypothetical example to the same treatment as that 
used by Pratt. 


Two subjects coincidentally made three pairs of 
similar drawings in five trials, the chance expectation 
of this occurrence being 1 in 11. When the two sets of 
§ drawings are submitted to 9 judges, they would pre- 
sumably each make 3 correct matchings, a total devia- 
tion of 18 beyond the chance expectation of 9 correct 
drawings. If the number of trials (n) is taken as the 
the number of items in each series (5) multiplied by 
the number of judges (9), then n = 45. Substituting 
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d= 18, n = 45, p = 1/5, and q = 4/5 in the formula, we 
obtain a critical ratio of 6.71, i.e., the probabili- 
ties of obtaining a deviation of 18 with an n of 45 are 
less than 1 in 1,000,000. Thus by multiplying the nun- 
ber of items in the series by the number of persons 
matching the two series, and by taking the total number 
of correct matchings it is possible to obtain almost 
infinitesimal chance probability values in a situation 
where the actual probability is 1 in 11. 


When the correct n for these data (5) is sub- 
stituted for the inflated n and the deviation in the 
sample (2.0) substituted for the total deviation, we 
obtain a critical ratio of 2.23, which permits a much 
closer approximation of our known probability of 1 in 
11. Similarly, if the correct n for Pratt's data (32) 
is used and the mean deviation obtained by the judges 
(5.0) is assumed to be the actual deviation in the sam- 
ple, his critical ratio of 5.3 becomes 1.77. The 
probability that the deviation would be obtained by 
chance is raised from less than 1 in 1,000,000 to ap- 
proximately 1 in 26. 


To summarize: We have pointed out that Chapman's 
statistics of the method of correct matchings give the 
probabilities of obtaining a given mean number of cor- 
rect matchings by a group of judges, and that Pratt's 
interpretation, which assumes this figure to be the 
probability of obtaining a given degree of resemblance 
between paired items in the two series, is in error. 

The fact that Pratt proposes the use of an inflated n 
was also demonstrated. 
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Note: Since this paper was written, Pratt's article has 
received further notice. It is referred to on pages 65 
and 84 of Fxtra-Sensory Perception after Sixty Years, 
by Rhine, Pratt, Stuart, Smith, and Greenwood. These 
references do not recognize the errors pointed out in 
the present paper, but in Appendix 10 (pp. 383-384 of 


4. This critical ratio of 1.77 would be lowered somewhat more were 
the variability of the judges' scores taken into account. 
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the same volume) the correct application of Chapman's 
statistic is presented (presumably by Greenwood). In 
addition, a paper by Greenwood (A caution in the use 
of the method of correct matchings. Am. J. Psychol., 
1940, 53, 614-615) points out the possibility of mis- 
interpreting Chapman's method and of making the error 
which Pratt made. However, reference to Pratt's paper 
is omitted. 

















COMMENT ON DR. ELLSON'S CRITICISM 


by 
J. G Pratt 


I appreciate the courtesy of Dr. Ellson and the 
editors of the Journal of Parapsychology in permitting me 
to publish simultaneously a brief statement about the 
foregoing article. This is not done to debate any of the 
fundamental issues Dr. Ellson has raised, but rather in 
the interest of correcting some unwarranted impressions 
which his discussion seems to me to create. 


Dr. Ellson says that I misinterpreted the Chap- 

man statistic. The fact is that my interpretation of 
the method was the very one that Chapman intended and 
that the "error" Dr. Ellson imputes to me alone is al- 
most as widespread in the psychological literature as the 
use of the method of correct matchings itself. As Dr. 
Ellson's note suggests, credit for discovering the ambi- 

ity likely to arise in the use of this method goes to 
r. J. A. Greenwood. The authors of Extra-Sensory Per- 
ception after Sixty Years were at that time well aware 
of the inapplicability of the Chapman statistic. If I 
interpret Dr. Ellson correctly, he wishes his readers to 
believe that we attempted to conceal the error in my 
paper. (This impression is confirmed by his specific 
emphasis of this charge in a review of our book in the 
Psychological Bulletin for December, 1940.) By way of 
reply I quote from page 65 of our book the first of the 
two references upon which he bases this contention: 
"Attempts to Fvaluate Non-Quantitattve Material. ... 
Pratt evaluated the results of Rice by means of the 
method of correct matchings, the statistical basis for 
which has been supplied by Chapman (as mentioned in Chap- 
ter II). The present status of the use of non-quanti- 
tative material as the basis of experiment seems to offer 
no fully satisfactory method of evaluating the results 
(but see p. 383)." age 383 criticizes the Chapman meth- 
od and sets the conditions for its proper use! I do 
meee how Dr. Ellson could have failed to get the con- 
nection. 
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LETTERS TO THE EDITORS 


Editorial Note: The editors again express their pro- 
found appreciation of the assistance given by the Board 
of Review, which continues to help in many ways in main- 
taining high standards of scientific reporting. Ina 
few cases, unfortunately, the cost of resetting material 
cannot be borne by the Journal's limited income, but in 
all essential points the Board's guidance has been grate- 
fully accepted. .s 


B. F. R. 


December 16, 1940 


Board of Review, Journal of Parapsychology 
Memorandum to the Editors. 


Subject: Review of Studies in Extra-Sensory Perception, 
III. A Review of All University of Colorado Ex- 
periments by Dorothy R. Martin and Frances P, 
Stribic. . 


This manuscript has been reviewed by the Board 
and the following members have submitted opinions which 
are included in this report: E. R. Hilgard, R. R. Wil- 
loughby, I. Lorge, J. L. Kennedy, L. Long, L. Dick and 
S. B. Sells. 


This paper was well received by the committee. 
Dr. Willoughby stated that there is a need for more of 
this kind of careful reporting. Dr. Long and Miss Dick 
felt that the authors should be complimented on the qual- 
ity as well as on the quantity of the experiments. 


Dr. Hilgard stated that "because of the obviously 
unsatisfactory conditions in the preliminary series, it 
is unfortunate to present grand totals in which better 
conditions are mixed indiscriminately with poorer ones, 
as in Table U.1i." 


Dr. ponge believed that the paper was written in 
good reportorial style and recommended its publication. 
He made the following detailed suggestions: that on page 
IV the expression in No. 5, “happy confluence...etc" be 
changed; in the introduction he would delete the line 
"Much of this.....to offer," and take out the word "never 
theless" two sentences away; On page 2 he would write 
"adequate supply of the special cards"; on page 4 he 
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suggests that the 55555 check should be explained; On 
page 11 the sentences immediately following the words 
general psychology should read, "group was composed of 
individual volunteers"; with reference to the table he 
recommends that the table heads be set up with No. of 
Runs, Total Hits, Average Hits per Run of 25. The last 
suggestion might be consistently adopted throughout the 
text. 


It is recommended that the above suggestions 
would improve the gi eaters of the experiments and should 
be incorporated in the published report. 


Saul B. Sells, 
Chairman. 
Department of Education, 
Brooklyn College. 


December 16, 1940 


Board of Review, Journal of Parapsychology 
Memorandum to Dr. Gardner Murphy, Editor 


Subject: Review of Variations of Time Intervals in Pre- 
Shuffle Card-Calling Tests by Lois Hutchinson. 


This manuscript has been reviewed by the Board 
and the following members have submitted opinions con- 
mg | it: E. R. Hilgard, R. R. a ay te I. Lorge, 
J. L. Kennedy, L. Long, L. Dick and S. B. Sells. 


The Board feels that certain revisions indi- 
cated in the discussion below should be incorporated in 
the paper prior to publication. It is suggested that 
or changes will make the report clearer and easier to 
ollow. 


Dr. Lorge has suggested that the report was clear- 
ly presented throughout, except for the conclusions. He 
feels that the paper would be strengthened if series 2 
and series 1 were defined in the conclusions. He also 
States that it is important to recognize that the empiri- 
cal distribution of hits is not normal binomial. Per- 
haps some more work should be done with the whole prob- 
lem of individual favoritism in runs or groups of runs. 


Dr. Long has made the same criticism concerning 
the distribution of hits. 
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Dr. Willoughby does not understand where the 
gelatin duplicator or hectograph comes in since the a 
card orders were recorded apparently in private by a perm 
son who could not help seeing the calls already made, ~ 
which were recorded beside the blanks where he was to © 
make his record. This should be explained. : 


Dr. Hilgard states that "the results are cau- 
tiously handled. The interpretation of probabilities ~ 
still permits of some choice. For example, the signifi- 
cant deviations of the one-day checked scores (the only 
ones for which significance is claimed) would occur only 
once in 1600 times as judged by the critical ratio of 
3.11, but would occur once in 30 times in terms of the ~ 
Chi-square treatment, as the probability values are pre-7 
sented, on page 10 of the manuscript." This point shoul@ 
be clarified in the final copy. 


[The duplicator saves time. It also helps i 
slightly in guarding against error, but is not the chief 
method described for this purpose, The hectograph helps 
to prevent post hoc "selection" of record sheets. 


It is agreed that the critical ratio method and | 
the chi-square method give different p-values. This is 


the general situation in experimental psychology, and | 
one of the reasons why a p-value as such is usually somes 
what arbitrary.--Ed.] 


Saul B. Sells, 
Chairman. 
Department of Education, 
Brooklyn College. 


























